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ABSTRACT 

The General Accounting Office (GAG) evaluated the 
Office of Economic Opportunity's (OEO) performance contracting 
experiment because of its potential impact in education* Performance 
contracting has been defined as an agreement bet%^en a local 
education agency^ such as a public school^ and a private educational 
firm, known as a learning system contractor* Payment tc the 
contractor depends on student achievement. OEO initiated a major 
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remedial reading and mathematics programs conducted by private 
educational fiinns. A total of 27,000 students, 18 school districts, 
and 6 private educational firms were involved* It was found that the 
firms operating under performance contracts did not perform 
significantly better than the more traditional school systems. 
(Author/CK) 
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To the President of the Senate and 

the Speaker of the House of Representatives 

This is our evaluation of the Office of Economic Opportunity's 
performance contracting experiment. 

We made our review pursuant to the Budget and Accounting 
Act, 1921 (31 U.S.C. 53), and the Accoimting and Auditing Act of 
1950 (31 U.S.C. 67). 

We are sending copies of this report to the Director, Office 
of Management and Budget; the Acting Director, Office of Economic 
Opportimity; and the Secretary of Health, Education, and Welfare. 




Comptroller General 
of the United States 
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EVALUATION OF THE OFFICE OF 
ECONOMIC OPPORTUNITY'S PERFORMANCE 
CONTRACTING EXPERIMENT B-130515 



DIGEST 



WHY THE REVIEW WAS MADE 

The General Accounting Office (GAO) 
evaluated the performance contract- 
ing experiment because of its poten- 
tial impact in education and because 
it was the Office of Economic Oppor- 
tunity's (OEO) first major experi- 
ment after its designation by the 
President as the primary research 
and development arm for the Nation's 
poor. 

Baokground 

"Performance contracting" has been 
defined as an agreement between a 
local education agency, such as a 
public school, and a private educa- 
tional firm, known as a "learning- 
system contractor." Payment to the 
contractor is related to some 
measure of student achievement. 
In other words, the contractor is 
paid on the basis of its success in 
raising the grade levels of students 
it instructs. Performance contract- 
ing is not a program but a method of 
organizing programs. 

Prompted by the initial reports of 
success of the first performance 
contracting project, OEO initiated 
a major educational experiment in a 
conscientious effort to help poor 
children because of its belief that 
education was crucial to breaking 
the poverty cycle and to provide 
useful information to the many 
school districts which were con- 
sidering such projects. 



The OEO experiment, conducted dur- 
ing the 1970-71 school year at an 
estimated cost of $6 million, was 
designed to assess the overall 
impact of remedial reading and 
mathanatics programs conducted by 
private educational firms. These 
programs were carried out under 
performance contracts for students 
from low-income' families performing 
well below average in the subjects 
relative to national norms. 

The experiment included approxi- 
mately 27,000 students, 18 school 
districts, 6 private educational 
firms, a management support con- 
tractor, a test and analysis con- 
tractor, and a payment computations 
contractor. (See app. II.) 

The President's fiscal year 1974 
budget contains no direct appro- 
priation to OEO and provides for 
the transfer of certain OEO pro- 
grams to other Federal agencies. 
Funds will be provided in the fis- 
cal year 1974 budgets of these Fed- 
eral agencies for continuing these 
programs. 



FINDINGS AND CONCLUSIONS 

Was performance contracting more 
successful than traditional class- 
room instruction in improving the 
reading and mathematics skills of 
poor children? The answer accord- 
ing to OEO is nol OEO's report 
released in June 1972 stated that 
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"The results of the experiment 
clearly indicate that the fions op- 
erating under performance contracts 
did not perform significantly 
better than the more tradUional 
school systems." 

Because of a number of shortcomings 
in both the design and implementa- 
tion of the experiment, GAO be- 
lieves that the question as to the 
merits of performance contracting 
versus traditional educational 
methods remains unanswered. (See 
p. 13.) 

While the OEO experiment in per- 
formance contracting was initially 
designed to make a reasonable com- 
parison between educational per- 
formance contracting and tradi- 
tional classroom instruction, GAO 
believes the information obtained 
from the experiment did not provide 
Q basis for making a reliable com- 
parison. (See p. 14.) 

Comparison of exper-imental and 
control groups 

Although there were 6 unique experi- 
mental instructional programs 
involved in the 18 school districts, 
OEO's overall conclusion concerning 
the merits of the instructional 
programs of the educational firms 
was based on its comparative analy- 
sis of achievement results between 
experimental and control groups 
aggregated for all 18 school 
districts. (See p. 14.) 

OEO intended to compare the indi- 
vidual experimental instructional 
programs with the traditional 
school programs by analyzing 
student achievement results om a 
school-district-by-school-district 
basis. As a result, OEO reported 
that there were some successes and 



failures among individual school 
districts, at least in certain 
grades and subjects. 

OEO cautioned that these individual 
school district results are less 
reliable than the aggregate results 
because of (1) smaller sample size, 
(2) apparently extraordinarily 
large or small gains of control 
groups, and (3) unknown effects of 
less than ideal testing conditions 
at some school districts. (See 
p. 14.) 

OEO designed the experiment to in- 
clude students in the experimental 
and control groups who were compa- 
rable in terms of initial achieve- 
ment levels and socioeconomic charac- 
teristics, such as race and family 
income. OEO's student selection 
procedures resulted in a majority 
of the experimental and control 
groups not being comparable in 
terms of initial achievement levels 
or race and family income. As a 
result, OEO's comparison of student 
achievement results on an aggregate 
basis may not reliably indicate the 
relative effectiveness of the 
educational firms" instructional 
programs. (See p. 15.) 

Control groups not monitored 

Although OEO attempted to maintain 
controls over experimental instruc- 
tional programs conducted by the 
six educational firms, there were 
no indications that such controls 
were ever contemplated for educa- 
tional programs of the control 
schools. As a result, there was no 
assurance that achievement results 
reported for control schools repre- 
sent the typical results in those 
schools and serve as a valid basis 
for comparison to achievement gains 
of experimental schools. (See 
pp- 19 to 20.) 
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Length of instructional periods 
not coordinates^ 

Neither OEO nor its management sup- 
port contractor sought to control 
the amount of class time for both 
experimental and control students* 
Large differences in instructional 
period length occurred which could 
have affected ultimate achievement 
results of the experiment. Ques- 
tions concerning achievement gains 
exist because instructional time-- 
an important factor affecting stu- 
dent achievement— was not uniform. 
(See pp. 20 to 22j 

Information on significant 
experimental factors not col l ected 

Although one of the primary reasons 
OEO selected the six educational 
firms was that each offered a dif- 
ferent instructional approach to 
helping academically deficient 
students, the firms continuously 
modified their instructional ap- 
proaches dur' ^ the school year. 
Wo one firm exactly the same 
instructiona .erials at all 
three of its school districts. 
These factors obscured information 
on the relative effectiveness of 
the diversified instructional ap- 
proaches originally sought by OEO 
and adoed to the confusion as to 
the primary source of achievement 
differences resulting from the 
experiment. (See pp. 22 and 23.) 



Lack of operational preparedness 

OEO*s evaluation and report on the 
results of its experiment indicated 
that lack of operational prepared- 
ness did not significantly affect 
the results. GAO believes, how- 
ever, that this Shortcoming did, 
in fact, adversely affect the 
experimental outcome. (See p. 
25.) 



The educational firms were hampered 
by unfamiliarity of some of their 
project administrators with the 
firms' instructional programs and 
by absence of curriculum materials 
during teachers training sessions 
and at the start of school. The 
majority of the firms' project ad- 
ministrators and teachers were hired 
specifically for this project and 
were therefore inexperienced in the 
use of the firms* instructional 
approaches. (See pp. 26 and 27.) 

The short time available during 
selection of school districts 
forced many districts to agree to 
participate without full knowledge 
of all the implications. Since 
negotiations took place during the- 
summer, most school personnel did 
not kna/ they would be involved 
until school opened and this caused 
many to view the project with ap- 
prehension. Personnel in several 
school districts were openly criti- 
cal of and hostile toward the edu- 
cational firms during the school 
year. (See pp. 27 and 28.) 

The test and analysis contractor 
was selected with less than 2 weeks 
remaining prior to the start of the 
school year in several school 
districts and, as a result, was not 
adequately prepared to test about 
27,000 students. (See pp. 29 
and 30.) 

Because of insufficient time, OEO 
apparently did not adequately ex- 
plore the feasibility of the cost- 
effectiveness measure proposed by 
trie management support contractor 
before contractually agreeing to 
its proposal* Later the proposed 
measure was found to be infeasible 
because nec^^.ary cost data was 
not avails' , from some school 
district ^nd educational firms to 
make cost-effectiveness comparisons. 
Costs incurred to obtain information 
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significantly exceeded the con- 
tractor's estimate because the 
cost data was not readily avail- 
able in the other school districts • 
Results of the exercise were of 
little value because program costs 
were not related to student 
achievement as planned. (See pp. 
30 to 33.) 

Tei^t a dninietration 

Although test publishers stressed 
that standardized test conditions 
were needed and specified other 
requirements to achieve valid re- 
sults, they did not provide infor- 
mation concerning the effects that 
poor testing conditions would have 
on test scores. 

OEO and the test and analysis con- 
tractor recognized that poor test- 
ing conditions encountered for cer- 
tain grade groups and schools could 
have introduced a bias to their 
analysis of the experimental in- 
structional programs* impact, 
particularly at the school district 
level. (See pp. 34 to 41.) 

The requirement for interim per- 
formance objective tests was 
impracticable because conditions 
necessary to insure reliable 
results could not be met within 
the limited time before the tests 
were given. As a result, the tests 
were virtually useless for evalua- 
tion and questionable as a basis 
for paying the firms. (See pp. 41 
to 43.) 

Cont'mot administration 

OEO's procedures in soliciting and 
evaluating proposals and in award- 
ing contracts generally did not 
result in effective procurement. 
OEO e^ Juated the proposals of the 
educational firms using criteria 



other than that specified in the 
request for proposals, in viola- 
tion of the Federal Procurement 
Regulations. 

Although the OEO contracting of- 
ficer made a determination that the 
six educational firms selected were 
financially responsible, at least 
two firms, in GAO's opinion, did not 
meet the conditions necessary to 
make such a determination. 
(See pp, 44 to 59.) 

Paymente t o the educational firms 

Final payments to the three educa- 
tional firms that had settled with 
OEO as of March 28, 1973, bore 
little, if any, relationship to the 
achievement of students they in- 
structed although this was to be 
the crux of the performance con- 
tracting concept. (See pp. 60 
arid 69 J 

On the basis of actual student 
achievement as measured by pretests 
and posttests, the educational 
firms earned an average of only 
33 percent of the total possible. 
OEO subsequr.ntly made a number of 
adjustments which significantly 
increased proposed payments to the 
firmf^. (See p. 63.) 

The most significant increases 
resulted when OEO (1) reimbursed 
all firms for so-called lost 
instructional time—about $845, OOC— 
and (2) dropped the incentive pro- 
visions of the contracts in favor 
of cost reimbursement for all 
grades in one school district and . 
for three grades in another school 
district—about $172,000. (See 
p. 60.) 

6A0 believes that OEO's adjust^ients 
in many cases go beyond the origi- 
nal language and intent of the 
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contracts, (See p, 60.) In mak- 
ing adjustments to the educational 
finrts* earnings, OEO recognized 
that conditions under which the 
fims conducted their instructional 
programs differed significantly 
aniong school districts and firms* 
In some instances, conditions had a 
detrimental effect on the firms' 
ability to perform; that is, to 
instruct the students and thereby 
raise their achievement levels. 
(See p. 60.) 

GAO believes that any conditions 
which adversely affected perform- 
ance of the educational firms may 
have also adversely affected the 
reliability of the results of 
the comparison between experi- 
mental and control programs • 
(See p. 60-) 



RECOMMENDATIONS AND SUGGESTIONS 

OEO's research and development 
activities in education will be 
transferred to the National Insti- 
tute of Education, Department of 
Health, Education, and Welfare. 
Observations contained in this 
report are expected to be of value 
to the Institute and local educa- 
tional authorities if similar 



experiments are conducted in the 
future* 

age:jcy^ Acr;nys aw u^'^J^i-^solve^^ 

OEO stated that its final report in 
June 1972 contained a comprehensive 
analysis of the results of the ex- 
periment and that many of the prob- 
lems pointed out in GAO's report 
were appropriately noted in its 
report. OEO believes that its re- 
port provides a useful perspective 
within which the overall perform- 
ance contracting experiment may be 
judged, 

MATTERS FOR CO fi SI D ERATION BY 
THE CONGRESS 

The Federal Government spends about 
$6.5 billion annually for educa- 
tional programs. Congress has ex- 
pressed concern as to whether the 
Federal expenditures have resulted 
in improved educational effective- 
ness. A number of educational 
experiments. Including 0EO"s per- 
formance contracting experiment, 
have been directed at testing and 
developing more effective educa- 
tional programs • The Congress 
should find this report useful in 
its consideration of legislation 
involving Federal educational 
programs. 
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CHAPTER 1 



INTRODUCTION 

Under authority of section Z5Z{a) of the EconoTiiic 
Opportunity Act of 1964, as amended, the Office of lcon--MO 
Opportunity conducted an educational experiment under ;v i 
forniance contracting during the 1970-/1 school yenr ,M ,ir. 
estimated cost of about $6 million. GEO initiated this majci 
educational experiment in a conscientiou.^ effort to help poor 
children because of its belief that education was crucial 
to breaking the poverty cycle ^.id to provide useful informa- 
tion to the many school districts which were considering suci. 
projects • The experiment was designed to assess the overall 
impact of private educational firms* remedial reading and 
mathematics programs upon students who were performing well 
below average in these subjects. 

On February 23, 1975, we submitted a draft of this report 
to the Director, OHO, for review and comment • OEO comments, 
which were received by letter dated April 6, 1973, are included 
as apj^endix I and, where pertinent, are incorporated in the 
appli':able sections of this report. Segments of the draft 
repor: applicable to the educational firms, the management 
support contractor, and the lest and analysis contractor were 
also sent to them for review and comment. As of April 2, 1975, 
written comments had been received from three educational 
firms and the two contractors. Their comments were considered 
in the preparation jf this report. 

PERFORMANCE CONTRACTING IN EDUCATION 

••Performance contracting" has been roughly defined as a 
covenant between a local education agency, such as u public 
school, and a learning-system contractor (a private educa- 
tional firm) in which payment to the contractor is related to 
s'^me measure of the achievement of the students in the learn- 
ing program. In other words, the contractor is paid on the 
basis of its HKUccess in raising the grade levels of the 
students it instructs. Performance contracting is not a pro- 
gram but a method for organizing programs. 

Although the performance-contracting concept dates as far 
back as 1862, its first application in the education of public 
school students in the United States was in late lt}69. During 
the fall of 1969 it was introduced in the Poitland, Oregon, and 
the Texarkana, Arkansns, school systems. 



The Portland public school system experimented with five 
very small, locally financed programs during the second half 
of the 1969-70 school year and the 1970 summer session. Little 
publicity had been given these programs and no results were 
available prior to the OEO experiment. 

The Texarkana school system initiated an experimental 
program during the 1969-70 school year directed toward re- 
ducing student dropouts. This program was funded primarily 
by the Department of Health, Education, and Welfare. The 
private educational firm conducting the program was to be paid 
on the basis of its success in raising student achievement 
levels above a minimum guaranteed level of one grade. Although 
initial reports on the results of the program in early 1970 
indicated success, the results were later determined question- 
able when it was learned that the firm was "teaching to the 
tests.'* 



STRUCTURE OF EXPERIMENT 



The OEO experirnent included 18 school districts, 6 private 
educational firms, a management support contractor, a test 
and analysis cont actor, and a payment computations contrac- 
tor. Each school district had a cost-no-fee contract with 
OEO. The six private educational firms were each assigned 
three school districts by OEO. Although fixed-price incentive 
contracts were signed between the firms and their assigned 
school districts, the major contract provisions were negotiated 
between OEO and the firms. The firms were to be paid on the 
basis of their success in raising the mathematics and reading 
grade levels of the students they instructed. 

OEO initially estimated that the maximum cost for the 
experiment would be $6.7 million. As of December 13, 1972, 
the estimate was about $6 million, as follows: 

Initial 
contract 

Contractors estimates 



Estimated 
final cost 
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School districts (18) $1,186,000 $1,193,000 

Educational firms (6) 4,371,000 3,099,000 

Management support contractor 526,000 547,000 

Test and analysis contractor 614,000 1,082,000 

Payment computations contractor 15,000 26,009 

* 

$6.710.000 $ 5.947.000 
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Appendix II lists in detail initial contract estimates and 
proposed or final payments. 

The school districts OEO selected were considered to have 
a hioh concentration of children from low-income families who 
were performing below grade level in reading and mathematics 
and were a reasonable representation of geographic locations, 
urban and rural settings, and ethnic backgrounds. Each educa- 
tional firm was to employ a separate instructional approach or 
technique. Three school districts, each with from 450 to 600 
students below grade level in reading and mathematics, were 
assigned to each educational firm. 

The experiment was to include approximately 27,000 students 
from grades 1, 2, 3, 7, 8, and 9 who were classified into 
four groups--experimental , control, comparison, and special 
program groups. The educational firms instructed the experi- 
mental groi ps in mathematics and reading. The control groups 
were to be in the same school districts but not in the same 
schools as the experimental students. These groups were to 
be comparable in grade-level decrements and socioeconomic 
factors such as race and family income. A comparison of 
student achievement between the experimental and control groups 
was to provide the primary measure of each programs* effec- 
tiveness. 

The comparison groups were in the same schools and classes 
as the experimental students except for the mathematics and 
reading classes. A comparative analysis between the experi- 
mental and comparison groups was to indicate any possible 
rub-off effect of the experimental programs on the comparison 
group. The special program groups participated in two other 
remedial mathematics and reading programs not sponsored by 
OEO. A comparative analysis between the experimental and 
special program groups was to indicate the relative effective- 
ness of other remedial education programs. 

The management support contractor was to help OEO plan 
the experiment's design, select participants, and develop 
and monitor onsite reporting procedures and to technically 
assist the school districts. OEO's contract with the manage- 
ment support contractor provided for payment on a cost-plus-a- 
fixed-fee basis. The test and analysis contractor was to 
test the students and independently evaluate the results of 
the experiment for OEO under a cost-plus-a-fixed-fee contract. 
The payment computations contractor was to compute final 



payments to the educational firms based on student achievement 
under a fixed-price contract with OEO. 

The primary measure of program impact was to be based on 
student achievement as measured by standardized nationally 
normed^ pretests and posttests. However, other information 
concerning the students, such as attitudes of students and 
parents toward the experiment; characteristics, training, 
and attitudes of instructional staff; and attitudes of school 
district decisionmaking staffs, was to be collected and 
analyzed to help explain the overall impact of the experiment. 
OEO intended that, at the end of the experiment, it would 
be able to discuss the impacts of the experimental instruc- 
tional programs at any given school district, for any given 
program, for each type of student, and for all students in 
the experiment. 

now WRRE PyWiMENTS BASED ON STUDENT 
ACHIEVE MENT TO BE COMPUTED ? 

m 

Student achievement, as measured by standardized nation- 
ally normed pretests and posttests, was to serve as the basis 
for computing a maximum of 75 percent of the amount which the 
firms could earn under their contracts. The remaining 25 per- 
cent was to be based on the results of five interim performance 
objective tests which were to be administered to the experi- 
mental groups during the school year. 

Payment based on pretests and posttests 

Each firm guaranteed a minimum grade level increase for 
each subject in each grade. The firms established a base dollar 
amount to be recovered only for each student who achieved 
the minimum guaranteed grade level increase. For each 
0.1 grade level increase achieved by a student above the 
guaranteed minimum, the firm would recover a set unit dollar 
amount in addition to the base amount. 

To illustrate, firm E has a fixed-price incentive con- 
tract for $252,000 with a school district. The maximum amount, 
75 percent, that could be recovered on gains in grade level 



^The test publisher establishes a standard of achievement 
based on test results of a sample of students representative 
of all students in a particular grade throughout the country. 
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achievement as measured by the pretests and posttests was 
$189, 000 . Firm E guaranteed a minimum 1.5 grade level in- 
crease for students in the secondary grades. The base amount 
to be recovered for students achieving this increase was 
$82.50 per subject per student. For each 0.1 grade level 
increase above the minimum in each subject, the firm was to 
recover $15 per student. Following are several examples of 
what the firm could recover on student achievement. 



Example 1 

Student scored a gain of 1.0 in reading and 0.8 in 
mathematics. There was no payment; the firm had guaranteed 
a 1.5 increase in both subjects. 



Example 2 

Student scored gains of 1. 
mat ics . Payment to the firm 
computed as follows: 

Unit 
gai-: 
Student above 
gain 1> 5 



Reading 1.5 
Mathematics 1.4 



5 in reading and 1.4 in mathe- 
on this student is $82.50, 

Payment for 
increase 
above 
Base minimum 
payment ( $15 per 0.1 ) Total 

$82.50 - $82.50 



$ 82.50 



Example 3 

Student scored gains of 2.0 in reading and 1.8 in mathe- 
matics. Payment to the firm on this student is $285, computed 
as follows: 



Reading 
Mathematics 



Student 
gain 

2.0 
1.8 



Unit 
gain 

above Base 
1. 5 payment 



.5 
.3 



$82.50 
$82. 50 



Payment for 
increase 
above 
minimum 
($ 15 per 0.1 ) 

$75 
$45 



Total 

$157.50 
127.50 



$285.00 
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The minimum grade level guarantee varied among the firms 
and ranged from 0,5 in certain elementary grades to 1.5 in 
grades 7, 8, and 9. Four firms varied their minimum grade 
level guarantees between the elementary grades and grades 
7, 8, and 9, and two firms guaranteed a minimum of 1.0 grade 
level gains for all grades. 

Payment based on interim 
performance objective tes ts 

The firms could recover up to. 25 percent of the contract 
amount on the basis of the students* attaining passing grade 
of 75 percent or more on each of the five interim performance 
objective tests for both reading and mathematics* Each firm 
established a unit amount that could be earned per student 
in each subject. For example, one firm established a unit 
amount of $52*50 per student in each subject if the student 
passed all five tests, or $10*50 for each test passed. 
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RESULTS OF P1;RF0R?IANCH CONTRACTING^ liXPliiUiMliNT 

V/as performance contracting inore successful than 
traditional classroom instruction in imp roving the reading 
and mathematics skills of poor children? The answer accord- 
ing to OEO is no! OEO*s report released in June 1972 stated: 

"The results of the experiment clearly indicate 
that the firms operating under perfornance contracts 
did not perform significantly better tlian the :nore 
traditional school systems. 

***** 

Thus while we judge this experiment to be a 
success in terms of the information it can offer 
about the capabilities of performance contractors, 
it is clearly another failure in our search for 
means of helping poor and disadvantaged youngsters 
to develop the skills they need to lift themselves 
out of poverty.'' 

Because of a number of shortcomings in both the design 
and implementation of the experiment, it is our opinion that 
the question as to the merits of performance contracting 
versus traditional educational methods remains unanswered. 
These shortcomings include: 

--The experimental and control groups were not 

comparable in initial achievement levels and socio- 
economic characteristics, such as race and family 
income, 

--OEO*s design did not provide for monitoring control 
groups, 

--OEO*s design did not call for coordinating the length 
of class instructional periods. 

--0E0 gathered little information on the overall effects 
of the experiment other than student achievement tests. 
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lack of operational preparedness due to insufficient 
leadtime to plan and implement the experiment adversely 
affected the experimental outcome. 

The experiment was initially designed to permit a 
reasonable comparison between performance contracting and 
traditional classroom instruction. However, the above short- 
comings in implementation, when combined with design problems, 
resulted in the lack of information to provide a basis for 
arriving at a statistically reliable overall conclusion or 
to explain the reasons for the success or failure of an ex- 
perimental program at any given school district. 

BASIS FOR CONCLUSIONS 

Comparison of experimental and control groups 

Although there were 6 unique experimental instructional 
programs involved in the 18 school districts, OEO^s overall 
conclusion concerning the merits of the instructional programs 
of the educational firms was based on its comparative analysis 
of achievement results between experimental and control groups 
aggregated for all 18 school districts. OEO intended to com- 
pare the individual experimental instructional programs with 
the traditional school programs by analyzing student achieve- 
ment results on a school-district-by-school-district basis. 
As a result of these analyses, OEO reported that there were 
some successes and failures among individual school districts, 
at least in certain grades and subjects. OEO cautioned, how- 
ever, that results from individual school districts are less 
reliable than the aggregate results because of (1) the smaller 
sample size, (2) the apparently extraordinarily large or small 
gains of control groups, and (3) the unknown effects of less 
than ideal testing conditions at certain school districts. 

A random assignment of schools as experimental and 
control within the participating school districts and a ran- 
dom selection of students within these schools was to be made 
by OEO in order to minimize the analytical problems which 
would result from noncomparable experimental and control group 
students. Random selection is the most desirable method of 
selection in order to achieve initial comparability of groups 
and to reduce the possibility of extraneous source explana- 
tions for experimental effects. OEO, however, selected the 
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most deficient schools within the school districts as 
experimental and attempted to select for control purposes 
those schools v:hich most closely matched the experimental 
schools. The most academically deficient students were thou 
assigned to fill the experimental and control groups withia 
their respective schools. 

Consequently, although OEO designed the experiment to 
include students in the experimental and control groups who 
were comparable in terms of initial achievement levels and 
socioeconomic characteristics, OEO*s student selection pro- 
cedures resulted in a majority of the experimental and control 
groups not being comparable in terms of initial achievement 
levels and socioeconomic characteristics. As a result, tie 
comparison made by OEO of student achievement results on an 
aggregate basis between the experimental and control groups 
may not reliably indicate the relative effectiveness of the 
educational finas* instructional programs. 

The significance of this initial mismatch is twofold. 



Previous educational research has shown that groups 
starting at different educational achievement levels 
would be expected to experience different rates of 
growth. Therefore, the unadjusted student achieve- 
ment rates would tend to be biased in favor of the 
group of students having the higher initial achievement 
levels. 



2. There is no known statistical procedure which can 
reliably adjust the student achievement results to 
eliminate the bias resulting from the mismatch in 
initial educational achievement levels. 



As shown in the table below, the results of the mathematics 
and reading pretests disclosed that the majority of experimental 
and control group students at the 18 school districts were non- 
comparable in terms of initial achievement levels. 
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Readin: 



Grade : 
1 
2 
3 
7 
8 
9 



Comparison of Pretest Scores 
for the 18 School Districts 
!Jo Experinental 
d ifference greater 



6 
3 
7 
8 
4 
_6 

34 



1 
4 
2 
1 
1 



Control 
greater 



10 
11 
9 
9 



11 



63 



Total 



ai7 
18 
IS 
18 

18 
ai2 

106 



Mathematics : 
Grade : 
1 
2 
3 
7 
8 
9 



Total 



6 
9 
7 
6 
5 
_8 

41 

75 



2 
1 
1 
3 
1 
J. 

_9 

18 



10 
9 
12 
8 

56 

119 



18 
18 
18 
18 

al7_ 

106 
212 



^Insuf f icient test data available from pretests to wake a.iy 
comparison for one scliool district. 



As shoun above the control group students had great sr 
pretest scores in mathematics and reading in 119 of 212 in- 
stances, or 56 percent, and the experimental group students 
had greater pretest scores in 18 instances, or 8 percent. 
Consequently, a total of 64 percent of experimental and con- 
trol groups were mismatched and therefore would be expected 
to experience different rates of achievement. 

Even when the students' pretest scores are aggre/^ated 
for the 18 school districts, the comparison shows that the 
initial achievement levels of the control group studemts were 
higher and the difference in achievement levels increased 
with each succeeding grade, as shown in the followirr table* 
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Mean Pretest Grade Equivalent Scores 
Aggregated Across All 18 School Districts 

Reading Mathematics 

Experi- Differ- Expe ri- Diffcr- 

Grade mental Control ence mental Control ence 



1 


(a) 


(a) 




(a) 


(a) 




2 


1.5 


1.6 


.1 


1.4 


1.4 


.0 


3 


2.1 


•« * 3 


.2 


2.2 


2.3 


.1 


7 


4.5 


5.0 


.5 


4.7 


5.1 


.4 


8 


4.8 


5.6 


.8 


5.4 


5.9 


.5 


9 


5.6 


6.4 


.8 


6. 0 


6.6 


.6 



^The test used for grade 1 does not convert to grade level 
equivalents. 

OEO also attempted to select experimental and control 
schools that were matched in socioeconomic characteristics, 
such as race. The test and analysis contractor's final re- 
port to OEO showed tha^ the racial compositions of the con- 
trol groups differed from those of the experimental groups 
in seven school districts and in four the racial compositions 
were unknown. 



OEO extensively analyzed the results both on a 
school-district-by-school-district basis and on an aggregate 
basis. OEO used complex statistical procedures in an attenpt 
to correct for the initial mismatch, but concluded that the 
unadjusted student achievement results were * * as unbiased 
as any of the more complex approaches." Although OEO reported 
that there were some successes and failures at individual 
school districts, it questioned the reliability of these re- 
sults because of the smaller sample size, the apparently ex- 
traordinarily large or small gains of control groups, and 
the unkno\m effect of less than ideal testing conditions at 
certain school districts. 

Therefore the primary basis for OEO's conclusion that 
there was no significant difference in the performance of 
experimental and control group students was its aggregate 
analyses of unadjusted student test scores. 

OEO reported the following gains ia grade level 
equiv? ^ ents. 



ERIC 



17 



Differences in Grade Level Equivalent Gains 
oT^xperimental and Control Groups 

Experimental Control 

group gain group gain Difference 

Reading : 
Grade: 

1 (a) (a) (a) 

2 .4 0.5 -.1 

3 .3 0.2 +.1 

7 .4 0.3 +.1 

8 .9 1-0 -.1 

9 .8 0.8 



Mathematics 



Grade: 

1 (a) (a) (a) 

2 .5 0.5 

3 .4 0.4 

7 .6 0.6 

8 .8 1.0 -.2 

9 .8 0.8 



readiness test, rather than an achievement test, was used 
as the first-grade pretest. There is no grade equivalent 
for the readiness test. 

OEO also analyzed and reported the aggregate results of the 
experiment in terms of raw numerical test scores. In 4 of 
the 12 grade and subject combinations, the differences be- 
tween the gains of the experimental and control groups were 
statistically significant. Nevertheless, OEO concluded that 
the statistical differences were not educationally signifi- 
cant when converted to grade level equivalents since none 
exceeded a 0.5 grade level gain difference, a standard OEO 
believed was an appropriate measure of educational signifi- 
cance. 

From the grade equivalent results, one can conclude that 
there was no significant difference between the educational 
system of the private firms and of the traditional school 
system. However, analyzing results in this aggregate manner 
does not provide information regarding the success or failure 
of each firm nor does it provide any insight into whether 
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perfornance contracting works better in some circumstances 
than others. Moreover, since these results are based on 
unadjusted test scores, they tend to be biased against the 
experimental groups because of the mismatch in initial achieve- 
ment level. 

Control groups not monitored 

Although OEO attempted to maintain controls over the 
operations of the experimental instructional programs con- 
ducted by the six educational firms, there were no indications 
that such controls were ever contemplated for the operations 
of the educational programs of the control schools. Conse- 
quently, the experiment did not provide assurances that the 
control schools used traditional instructional methods nor 
were safeguards instituted to preclude teachers in control 
schools from inflating achievement gains by teaching to the 
tests or otherwise changing the traditional instructional 
program. For example, the management support contractor 
reported that the control school in one school district had 
a special education program for its deficient students. As 
a result, there was no assurance that the achievement results 
reported for control schools represent the typical results 
to be expected in those schools and serve as a valid basis 
for comparison to achievement gains of experimental schools. 

OEO*s management support contractor reviewed and monitored 
the activities of the educational firms to insure that the 
firms did not use curriculum items in their programs that 
could be considered directly related to items to be included 
on the achievement tests. However, similar precautions were 
not taken for control schools. The only information system- 
atically collected on control schools related to time spent 
in various activities by the in' .uctional staff. 

A number of speculations resulted from a lack of 
knowledge as to what was occuring at the control schools as 
indicated by the following comments from OEO^s report on the 
overall results of the experiment: 

It* * 1^ careful inspection of the site-by-site 
results suggests that at some sites, experimental- 
control differences might have resulted from extra - 
ordinarily large or small gains of the control 
rather than the experimental group." 
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* * * * * 



"A comparison * * * enables us to identify the cases 
Khere large positive or negative experimental-control 
differences ma^ be a result of abnormally small or 
large gains on the part of the local control group 
or site measurement error for the control students. 
We stress the word "may" because a gain score at a 
particular site which appears to be abnormally large 
or snail in relation to the av<»rage gain score across 
all sites may not be abnormally large or small for 
that particular site. Since we do not have the data 
to ascertain the degree to which "normal" gains vary 
from site to site, cases where there are differences 
between the pooled and unpooled adjusted mean gain 
differences are difficult to interpret." 



"Thus abnormally large or small control gains may 
also be obscuring a few cases of relatively good or 
bad performances of a contractor." 



"* * * some of the largest appar«;nt winners or losers 
^ be artificially inflated because of either control 
student volatility or control measurement erro r 
problems." ~~~ 



"There were some successes and failures among the 
individual sites, at least in certain grades and 
subjects, but even many of these are statistically 
quite unreliable - possibly caused by the volatility 
of the control students or site-wide testing diffi- 
culties." (Underscoring supplied.) 

Length of instructional periods not coordinated 

Neither OEO nor its management support contractor 
sought to control the amount of class time for both experi- 
mental and control students. Large differences in the length 
of instructional periods occurred which could have affected 
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the ultimate achievement results of the experiment. Also 
times available to the firms were 7iOt the same in all school 
districts. Questions concerning achievement gain.-; exist 
because instructional time — an important factor affectinv; 
student achicvement--was not uniform. 

Though OEO did not consider the length of daily class 
periods significant, research findings indicate that a re- 
lationship between achievement and the length or a class 
period does exist. 

Research, in general, indicates: 

--Children in primary grades do poorer with longer 
class periods because of eye fatigue and shorter 
attention spans. 

--Children at the secondary level do better with longer 
class periods. 

--Longer class periods are better for the study of 
mathematics than for reading. 

--Difference in length of class periodr. of from 10 to 
30 minutes can produce achievement differences for 
certain target groups and subjects. 

The an^ount of divergence in class instructional time 
is indicated by the following information in the managenent 
support contractor's final report to OEO on 10 school dis- 
tricts. The report did not include class instructional times 
for the remaining eight districts. 



Number of class hours 



Reading Mathematics 



Experi- 
mental 



Experi- 

Control mental Control 



Grades 1 through 3: 
School district: 



1 
2 
3 
4 
5 



1.156 
1.500 
0.750 
1.000 
0.920 



1.050 
2.000 
1 . 700 
1.083 
1.100 



1.156 
0.500 
0.750 
1.000 
0.920 



0.G67 
1.083 
0.927 
0.7S0 
0.833 
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Nunber of 


class hours 




R 


eading 


Mathematics 




Fxperi 


~ 


I:xperi- 






mental 


Control 


mental 


Control 


Graaes 1 through Sr 










School district: 










/: 
0 


0.750 


2 .000 


0. 750 


1.000 


7 


1 . 000 


1 . 546 


1.000 ■ 


1.028 


o 
o 


0.917 


1.917 


0.917 


0.667 


9 


0. 694 


1. 000 


0. 722 


1. 520 


lU 


1 . 000 


1 . 500 


1.000 


1.000 


Grades 7 througli 9 : 










School district: 










1 


0.806 


0. 917 


0.806 


0.917 


2 


0.917 


0.917 


0.917 


0.917 


3 


1. 000 


0. 717 


1.000 


0. 717 


4 


0.830 


0. 833 


0.840 


0.833 


5 


0. 750 


0.833 


0. 750 


0.833 


6 


0. 750 


1.000 


0. 750 


1.000 


7 


1.000 


0. 917 


1.000 


0.917 


8 


0. 91/ 


0. 726 


0.917 


0.725 


9 


0.917 


0. 889 


0.917 


0.889 


10 


0. 786 


0. 750 


0. 786 


0.750 



Only school district 2 had instructional periods for 
experimental and control groups that were exactly the same- 
but only for grades 7 through 9. Hoivever, class times for 
several other districts were, for all practical purposes, 
equivalent for both experimental and control groups. The 
general trend, however, was a divergence in time between 
experimental and control groups with some control classes 
having more than twice as much time as experimental classes. 

Information on significant experimental 
factors not collected 



Although one of the primary reasons OEO selected the 
six educational firms was that each offered a different in- 
structional approach to helping academically deficient 
students, the firms continuously modified their instructional 
approaches during the school year. The school districts' 
project directors reported to OEO that, after the programs 
had been implemented and had been operating, some firms found 
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it necessary to modify their programs and that they often 
changed materials and supplies. 

The project directors emphasized that, as the programs 
progressed, it appeared that each firm did not nave an in- 
dividual or a unique curriculum and that almost all firms 
used similar core materials. Moreover, no one firm had 
exactly the same instructional materials at all three of its 
school districts. These factors obscured information on the 
relative effectiveness of the diversified instructional ap- 
proaches originally sought by OEO and added to the confusion 
as to the primary source of achievement differences resulting 
from the experiment. 

OEO also initially planned to document a number of 
other major interest factors as part of its overall analysis 
of the impact of the experimental programs. These factors 
included student scores on payment tests, interim performance 
objective test scores, report card grades in other subjects, 
and changes in the attitudes of parents and school decision- 
makers • 

The test and analysis contractor was also to interview 
approximately 50 ninth grade students in the experimental 
groups, 50 in the control groups, and 50 in the comparison 
groups at each school district. The interviews were to be 
made at the end of the school year and were to assess atti- 
tudes toward school, perceptions of schools and teachers, 
feelings toward the experimental program, etc. These inter- 
views, however, were not conducted because OEO dropped the 
requirement . 

Changes in parental attitudes were to be measured by 
questionnaires filled out by parents at both the beginning 
and end of the school year. The first questionnaire was to 
include questions related to socioeconomic data to be used 
for insuring a close analytical match between experimental, 
comparison, and control groups, and the second questionnaire 
was to omit the socioeconomic questions. 

OEO was unable to assess the changes in parental 
attitudes because only one questionnaire wac sent to parents 
during the school year. OEO dropped most of the requirements 
for collecting and analyzing the remaining information during 
the school year for a number of reasons, not the least of 
which were the cost overruns. 
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la our opinion, the usefulness of the comparative 
achievement results was impaired also by the general lack of 
operational preparedness of the various contractors and the 
less than ideal testing conditions in certain grades and 
groups. These two factors are discussed in detail in the 
following chapters. 



ERLC 



24 



CHAPTER 3 



LACK OF OPERATIONAL PREPAREDNESS DUE TO 

INSUFFICIENT TIME TO PLAN 

AND IMPLEMENT EXPERIMENT 

OEO*s evaluation of and report on the results of its 
experiment indicated that the lack of operational prepared- 
ness did not significantly affect the results. We believe, 
however, that this shortcoming did, in fact, adversely af- 
fect the experimental outcome. 

The test and analysis and management support contractors, 
the 6 educational firms, and the project directors at the 18 
school districts all expressed the opinion that the experi- 
.nent uas hampered by insufficient startup time. As a re- 
sult of the short startup time, a great deal of confusion 
and disenchantment with the experimental education programs 
was evidenced. Neither the school districts, the test and 
analysis contractor, nor the educational firms were ade- 
quately prepared to begin their tasks within the extremely 
limited time available or to cope with the ensuing problems 
they encountered. 

V' e test analysis contractor was selected with only 
about 2 weeks remaining prior to the start of the school 
year in many school districts and, as a result, was not 
adequately prepared to test 3Cbout 27,000 students. 

Further, because of insufficient time, OEO apparently 
did not explore the feasibility of the cost-effectiveness 
measure proposed by the management support contractor prior 
to contractually ^Lgreeing to its proposal. Later the pro- 
posed measure was found to be infeasible because the neces- 
sary cost data was not available from some school districts 
and firms to make cost-effectiveness comparisons. In addi- 
tion, the costs incurred to obtain the information signifi- 
cantly exceeded the contractor's estimate because the cost 
data was not readily available in the remaining school dis- 
tricts. The results of these efforts were of little value 
because program costs were not related to student achievement 
as planned. 
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EDUCATIO\\AL FIRMS 



Four firms stated in a joint paper to OEO that:, 

" * * * from its inception, elements of the exper- 
iment were so poorly conceived and conducted * * * 
that these deficiencies should raise serious ques- 
tions within the educational community on the 
broad generalized conclusions released by OEO. 
* * * It is conservatively estimated that the 
first 4 months were devoted to reaching the nor- 
mal September status for experimental students.'' 

Project directors in the 18 school districts reported 
to OEO that the short leadtime adversely affected the ex- 
perimental instructional programs. They reported also that 
the preservice training of project staffs was hampered by 
the unfamiliarity of some educational firms' project adminis- 
trators with the instructional program and by the absence 
of most materials and equipment to be used during the train- 
ing workshop. The project directors stated that "all of 
these problems center around a lack of sufficient time." 

The majority of the firms' project administrators and 
teachers were hired specifically for this experiment and 
were therefore inexperienced in the use of the firms' in- 
structional approaches. The project directors reported that 
one of the major deterrents to the educational firms' being 
prepared to implement their programs at the start of the • 
school year at various school districts was the ill timing 
in hiring project administrators. Though some were employed 
prior to negotiating and finalizing subcontracts, others 
were not employed until a few days before the start of 
school or after the project had been implemented. This delay 
primarily caused the inadequate preservice training given to 
educational firm personnel. 

Preservice training for educational firm personnel 
ranged from 3 days to 2 weeks, depending on the firm and 
school district. Even at the school districts for which a 
greater number of training days were available, the proj- 
ect directors stated that the effectiveness of training was 
severely hampered by the lack of materials to be used for 
demonstration and practice. The educational firms' full 
complement of instructional materials or equipment was not 
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at the sites at the start of the school year for at least 
9 o£ the 18 school districts. At one school district, 
for example, the management support contractor reported 
teachers to be ''scrounging^' materials from the school dis- 
trict. 

S CHOOL DISTRICTS 

01:0 select-ed the 18 school districts between May 26, 
1970, and June 19, 1970. The school districts were given 
3 days to complete questionnaires which served as the basis 
for selection. At a 1-day conference between OEO and rep- 
resentatives of the school districts, the school districts 
and educational firms were matched, the intent and structure 
of the experiment was explained, major contract provisions 
were agreed upon, requirements for managing the experiment 
at the local level were determined, and each district's ad- 
ministrative costs were determined. Since these events all 
took place within a space of about 3-1/2 weeks when most 
schools are in i^ecess for summer vacation, the school dis- 
tricts had little or no time in which to solicit support 
from school principals, teachers, parents, or local unions 
or associations. 

After the experiment the project directors reported to 
OEO th?,t the 3hort time available during the schcol district 
selection process forced many school districts to make a 
"go-no go" decision without full knowledge of all the impli- 
cations. The project directors stated that the planning and 
implementation problens associated with the short leadtime 
caused many teachers and local school administrators to view 
the project with apprehension. An official of the manage- 
ment support contractor informed us that the school person- 
nel's initial impression of the experimental programs was bad 
because of insufficient* startup time and that this impression 
persisted throughout the school year. 

Teacher strikes hampered tM experimental projects at 
four school districts, and four others were under consider- 
able pressure from teachers* unions and school officials op- 
posed to the experimental programs. This opposition had a 
detrimental effect on the experimental program operations at 
these school districts and resulted, in part, because the 
school districts did not have sufficient leadtime to conduct 
an effective community relations program, especially at the 
outset of the school year. 
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The test and analysis contractor's pretest monitors 
reported that school principals and/or teachers at 8 
school districts were not adequately informed of the pur- 
pose or intent of the experimental programs and that at 2 
of these school districts even the project directors did not 
fully understand program requirements. For example, the 
project director at one school district was not aware of the 
requirement for testing comparison group students at the time 
the test monitor arrived on the site to begin pretesting. 

Many of the school districts which experienced one or 
more of these difficulties also reported instances in which 
teachers and school officials questioned the effectiveness 
of the project; teachers were frustrated and morale was low; / 
and parents either refused to allow their children to partic- 
ipate in the project or requested that those already enrolled 
be removed, which reduced the number of experimental group 
students. At least two school districts threatened to with- 
draw from the contract because the educational firms were 
not living up to expectations. Five other school districts 
threatened contract termination because of conflicts between 
the firms and school districts or alleged contract noncompli- 
ance. 
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TEST AND ANALYSIS CONTRACTOR 



The test and analysis contractor was to pretest all 
students within the first 10 days of school. But since 
the contractor was not authorized to incur costs until 
August 17, 1970, and school opened in August in six school 
districts and in September in the others, the test and 
analysis contractor had very little time to procure tests, 
blank out publishers' identifying information on the tests, 
ship them to the school districts, and organize and im- 
plement testing for about 27,000 children. 

In critiquing the performance of the test and analysis 
contractor, the 18 project directors reported to 0130 that 
a lack of adequate preplanning time for determining testing 
sites, selecting students, selecting and adequately train- 
ing testers, and preparing test booklets created hardships 
on the local school districts. The project directors 
stated that the test and analysis contractor's demands 
were not made known soon enough and that the test and 
analysis contractor's representatives arrived at the school 
districts without a clear understanding of their roles and 
responsibilities regarding the testing program. They stated 
further that some contractor representatives lacked ex- 
perience in mass-testing and sone school districts lacked 
the ability to handle mass- testing. 

The test and analysis contractor informed us that all 
of its representatives were professionally qualified to 
coordinate mass- test ing of students and that the extensive 
set of materials and the 1-day training session it conducted 
permitted its representatives to arrive at the school 
districts with a clear picture of their roles and responsi- 
bilities. Of the 17 representatives that visited the 
school districts, 5 had doctor's degrees and 12 had the 
required master's degree. The contractor stated further 
that, in fairness to all, the organization of the pretest- 
ing phase was a difficult assignment and the time frame 
was too short. 

The test and analysis contractor did not supply the 
pretest results to some districts until late fall. The 
project directors stated that this made it impossible to 
identify pretested students and that pretests sometimes 
had to be given to some students as late as January 1971. 



The test and analysis contractor informed fs that there 
were relatively few cases in which pretesfj were adiain- 
istered late and that the majority of the pretest results 
were made available to the districts by October 1970. 

MANAGEMENT SUPPORT CONTRACTOR 

Because of the insufficient leadtim-a OEO apparently 
did not adequately explore the feasibility of the cost- 
effectiveness measure proposed by the management support 
contractor before contractually agreeinjj to the contractor's 
proposal. Later the contractor's proposal \<as found to be 
infoasible because cost data was not available frora some 
school districts and educational firms. The scope of the 
cost analyses had to be reduced signf icantly during the 
year, but the costs associated with the analyses and billed 
by the contractor to OEO increased. The cost comparisons 
between the experimental and control programs per school 
district were of little value because the costs were not 
related to student achievement gains as contemplated in the 
contract requirements. 

OEO's contract with the managenent support contractor 
required the contractor to develop a cost data system 
which would identify program component costs per unit of 
student achievement and compare these costs for each in- 
structional approach. The estimated cost of this task was 
about $20,000. 

The management support contractor obtained the cost 
data from several data collection forius completed by school 
district officials or by the managemeit support contractor 
during its onsite visits. This information related to the 
resources consumed by the program for such things as build- 
ing space and teachers' salaries. 

All direct costs related to the specific instructional 
program, plus a prorated share of the costs of all other 
activities that either involved or supported the student, 
were to be allocated to each instructional program. The 
supportive or noninstructional costs were allocated among 
the academic subjects on the basis of time spent in each 
subject. Experimental and control programs often differed 
in the amount of time required, pattern of resources con- 
sumed, and consequently in the amount of supportive costs 
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allocated to them, OEO and educational firm costs not 
typical of traditional school operations, such as adminis- 
trative costs of the firms, were excluded, since OEO's 
intent was to estimate the cost that would be incurred if 
the programs were incorporated into the regular school 
programs • 

The task of gathering the required cost data fell be- 
hind schedule shortly after the start of the school year, 
and the situation became progressively worse. By March 1971 
OEO dropped attempts to accumulate costs for six school 
districts since OEO received little or no information fron 
these school districts. Because insufficient data was 
provi^ied by the educational firms, attempts for two addi- 
tional districts were subsequently dropped. 

Although the scope of the cost analyses was reduced sig- 
nficantly, the cost to accumulate the data and make the nec- 
essary analyses rose from the initial estimate of $20,000 to 
an actual cost of about $57,000. 

Because the thoroughness of the manner in which data 
was collected and substantiated varied frpjn district to 
district, the project directors believed^he use of the cost 
data system to evaluate and compare programs and program 
costs was questionable. They stated that this variance was 
probably due in part to the lack of cooperation by local 
educational personnel as well as the unavailability of data. 

Further, the management support contractor's cost 
comparisons between the experimental and control programs 
were distorted because the comparisons included noninstruc- 
tional costs. Noninstructional costs were allocated to the 
experimental and control programs on the basis of class 
time. However, many of these costs, such as salaries of 
administrative personnel and permanent plant and equipment 
costs, would not vary with the amount of class time. For 
example, the cost to operate a cafeteria does not depend 
on classroom time and therefore would not be an important 
consideration in comparing costs of different instructional 
techniques . 

We believe that only direct instructional costs should 
be considered in determining the incremental or additional 
costs associated with selecting alternative programs. When 
noninstructional costs are included, the fundamental dif- 
ferences in the cost structures between the experimental and 
control programs are distorted. 
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For exanple, in its final report to OEO, the management 
support contractor showed the following comparison of the 
per student year costs between the experimental and control 
programs in reading for the elementary grades in one school 
district. 

Total cost per 
Program student year 

Control $216.63 
Experim.-?ntal 186.47 

It appears that the control program is more expensive 
than the experimental. However, the control program 
spent 1.5 hours per day on reading instruction while the 
experimental program spent only 1 hour per day. Since 
nonmstructional costs were allocated on the basis of time 
in class, 50 percent more was allocated to the control 
program than to the experimental program. Consequently, 
by eliminating the noninstruc tional costs from the compar- 
ison, the experimental program becomes more costly than 
the control program. 

Instructional 
Program cost only 

Control $118.88 
Experimental 123.23 

By eliminating the nonins tructional costs from the 
comparisons, only a few programs changed from being more 
costly to less costly. However, any differences in the 
instructional times between the experimental and control 
programs would cause the cost comparisons to be less use- 
ful to local school officials for determining the dif- 
ference in cost which could be expected if the alternative 
program was adopted. 

In its final report to OEO, the management support 
contractor did not break down the instructional and non- 
instructional components of the total cost. Consequently, 
OEO did not attempt to determine to what extent noninstruc- 
tional costs allocated to the programs had distorted the 
cost comparisons. 
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After we brought this matter to OlO'r attention 0] 0 
obtained the necessary cost breakdown fro:n t!ic iiK.n.i.oenont 
support contractor and included it in its final report in 
•June J972. 

Furthermore, because of the unreliability of the 
student achievement data on a school-d i s tr Lc t"-by- >;choo I - 
district basis, OEO was unable to make the cost-ef fccti vcnioss 
comparisons initially intended, that is, the cost per unit 
of student achievement. Because an accurate measure of each 
program's cost -effectiveness could have been made only in 
this manner, OnO's comparisons of total costs between prcrar^ 
at each school district are of little value. 
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ADMIN] ST RAT IPX 01- VV.]:i 1 ,^ rs AND POSTTl ^STS 

AND INTERIM PHRPORMANCH OlMrCTIVi: TliSTS 

Student achievement, as measured by standardized 
nationally normed pretests and posttests, was to be the pri- 
mary measure of the impact of tl;c experiment, a? well as a 
basis for up to 75 percent of final payment to the educa- 
tional firms. Interim performance objective tests were to 
be used as a basis for up to 25 percent of the final payment 
to the educational firms and as a suppl eir.entaj measure cf 
the impact of individual educational firms' instructional 
programs . 

Although the test publishers stressed that standardized 
testing conditions were needed and specified other require- 
ments to achieve valid results, they did not provide informa- 
tion concerning the effects that poor testing conditions would 
have on test: scores. GEO and the test and analysis contrac- 
tor recognized that the poor testing conditions encountered 
for certain grade groups and schools could have introduced 
a bias to their analysis of the experimental instructional 
programs' impact, particularly at the individual school 
district level. 

GEO analyzed the aggregate results of the test data, 
however, on the assumption that the bias affected both th^ 
experimental and control groups equally. Reports on test- 
ing conditions prepared by the test and analysis contractor's 
onsite test monitors contained indications that thir assump- 
tion may not be valid. In many instances, however, the re- 
ports lacked sufficient information to determine the extent 
and seriousness of the poor testing conditions. 

Many of the less than ideal testing conditions during 
the pretests and posttests could be directly attributed to 
student behavior and may have occurred regardless of the 
amount of preplanning. However, many of the poor testing 
conditions resulted from a lack of planning and preparation. 

The requirement for interim performance objective tests 
was impracticable because the conditions, necessary to Insure 
reliable results could not be met within the limited time 
before the tests were given. As a result, the tests were 
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virtually useless for evaluation and questionable af a basis 
for paying the fi-.ns. 

TEST ADMINISTRATION DESIGN 

The test publishers provided specific instructions on 
how to administer their tests to insure vali.I and reliable 
scores. The publishers stressed the need for uniform test- 
ing conditions and set out certain specific requirements, 
including good lighting, freedom from crowding, adequate 
writing space, and a sufficient number of proctors. 

OEO's test and analysis contractor also specified two 
minimal conditions for administering the tests'. 

1. All tests were to be administered in classes of 

3S or fev/er students for grades 2 and 3 and classes 
of 2S or fewer students for grade 1. 

2. All junior high students were to nave tes ad- 
ministered in classes of 100 or fewer student; 

;ith one proctor for every 50 students, in addition 
to the test examiner. 

The test and analysis contractor stated that the recom- 
mended class sizes we -e established to insure standard condi- 
tions, to preclude cheating, and to spend test administration. 

Students were given two sets of standardized pretests 
and posttests. Both the experimental and control groups werj=' 
given an evaluation test at the beginning and end of the 
school year to ar.sess the overall impac* of the experiment. 
Only the experimental students were give.n a second set of 
standardized pretests and posttests to compute payments to 
the firms. Each grade was given three payment tests, except 
grade 1, which was given only one. The tests wore randowly 
assigned, eacn to one-third of the students, so that each 
student took only one test. 

The evaluation test called for two consecutive morning 
sessions for the elementary grades and two consecutive morning 
sessions plus one test administered on two consec.uti«?e after- 
noons for the junior high grades. The payment tests called 
for two consecutive morning sessions for the elementary graies 
and one morning session plus one separately timed cest in the 
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afternoon for the junior high grades. These test schedules, 
the test and analysis contractor stated, were designed to 
eliminate the effects of fatigue, limited spans of attention, 
and boredom, especially in the elementary arades. 

The test and analysis contractor, during a 1-day session, 
briefed test monitors for the pretesting. Emphasis was given 
to establishing a consistent pattern of operation among the 
school districts to provide as much standardization in adminis 
tering the tests as possible. In addition, the test monitors 
were to provide a 1-day training session for test examiners 
at the sites to insure standardization of test administration 
procedures. 

Each test monitor had to prepare a report answering 11 
questions on the test, covering such topics as student selec- 
tion, problems in arranging physical facilities and working 
with school personnel, test security, and events endangering 
valid test results. 

PREEXIST ADMINISTRATION 

The test and analysis contractor reported that, because 
of time constraints, it could not visit 10 school districts 
that opened by the first week in September to arrange for 
administering the test. Instead, they explained the testing 
requirements to these school districts by telephone. The 
test monitors arrived at the sites 3 days before the evalua- 
tion test was to be given. 

On the basis of the test monitors' reports, the test 
and analysis contractor reported to OEO that there were 
numerous deviations from its standardized testing procedures 
and incidents potentially affecting the validity of test re- 
sults . 

Deviations from standardized 
test administration procedures 

Student selection, testing facilities, and testing time 
schedules deviated from the standardized test administration 
procedures established by the test and analysis contractor. 
Test monitors at eight school districts reported that students 
and, in some cases, schools had not been selected when they 
arrived at the sites. This delayed the evaluation and payment 
testing. 



School districts had difficulty coupilinj; definitive 
lists of students to be tested for such reasons as hi^^li 
student turiiover ratvs and delays in obtaining parent'il vc:- 
pussion* Also, some control schools had not been ^^electeu. 
and scliools selected did not have enou<^h students for tiie 
exj^erimental and/or comparison groups and the control j^rruvs. 

The test nonitors reported that the class sizes durinj: 
testing for certain grade groups at seven school districts 
exceeded the maxiniuTP sizes established by the test and analy- 
sis contractor. For example, at one school district, elemen- 
tary school students from two or three grades were tested 
s Miltaneously in one large room. At another school district, 
tu grades were tostod I;. t*ie auditorium using lapboards as 
writing surfaces. Test n.onitors at the remaining 11 school 
districts either did net describe the facilities or reported 
that the physical facilities were satisfactory. 

Even though some problems were corrected after the 
first day*s testing, in some instances more than one grade 
was still tested in a room. The obvious problems of having 
more than oae test examiner speaking at a time and dissimilar 
testing time requirements for the various sections of the 
tests were compounded during the payment test since three 
different tests were given to each grade. Consequently, there 
were three different tests administered at one time in even 
those rooms wirh only one grade. 

Nine school districts had some difficulty adhering to 
the test schedule, mostly due to problems with student selec- 
tion and physical facilities. Other factors, such as weather 
conditions, were also reported to have caused delays. 

Conditions potentially affecting test validity 

Test monitors at 10 districts reported conditions which 
could have affected the validity of the pretest results for 
certain grades and/or groups. At two of these school dis- 
tricts, certain grades and/or groups had to be retested to 
obtain valid data. At one school district the evaluation 
test was regiven in grades 7, 8, and 9 because of discipline 
problems, inadequate testing facilities, and difficulty of 
students in hearing test administration directions in the 
facility used. At the other school district, extensive re- 
testing was involved because of the chaotic circumstances 
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surrounding the initial pretest. The monitor at this school 
district reported that the initial pretest was discontinued 
because ; 

--Students had not yet been selected when he arrived, 
teachers at one control school were on strike, and' 
another control school was outside the school dis- 
trict . 

--The project director was not aware that a comparison 
group had to be selected and tested. 

--The project director's staff was not yet hired. 

--The program had been given no publicity; principals 
had been notified that they were to participate but 
had not been told what the program was about. 

--Test facilities and other needs had not been discussed 
before he arrived. 

--Test facilities were inadequate. 
--Monitoring was inadequate. 

--Unions, school officials, and teachers opposed the 
project . 

--Testing conditions in all but grades 1, 2, and 3 of 
the control school were termed "mass confusion" with 
widespread cheating and student disinterest. 

The students were retested in mid-October- and the re- 
sults were considered satisfactory except that 'the control 
groups for grades 7, 8, and 9 were too small. Only the 
evaluation test was given in grades 2, 3, 7, 8, and 9 and 
only the payment test was given in grade 1. Consequently, no 
evaluation results were available for grade 1. 

At the remaining eight school districts where incidents 

were reported, problems most often centered arotind discipline 

m the classroom and were generally limited to the junior 

high school level. In several cases, only some of the grades, 
schools, and testing groups were affected. 
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In one of the more serious incidents, the test njonitor n' 
one school district reported that conditions anong the various 
testing locations varied greatly. Some rooms were air condi- 
tioned; others were not. At some locations the students ^■<-io 
much more unruly, disinterested, and unmotivated than at 
others, although a lack of motivation was evident at all 
locations. In the elementary grades, the payment and evil na- 
tion tests were given in 1 day each instead of the 2 days 
recommended by the test and analysis contractor. 

The test monitor reported that some students simply 
marked answers in a purely random pattern without regard to 
the questions. Some students slept through a considerable 
part of the test or talked and annoyed their neighbors. He 
stated that he could verify only the existence of test scores 
and could not certify that they represent a true measure of 
each student* s capability. He expressed concern over the 
assumptions that would be made from pretests to posttests on 
the comparability of the conditions under which the two sets 
of tests were administered, since he stated that there was 
no way that the conditions of the first test could be re- 
peated for the posttest. No retesting was conducted at this 
school district. 
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POSTTbST ADMINISTRATION' 



The test and analysic contractor jnade several 
reconunendations to OEO for improving the posttesting program 
by considering the difficulties encountered in pretesting which 
resulted from the minimum of time available for plaiining and 
coordinating the program among the many distinctive school 
districts. For the most part, the test monitors' reports 
showed that posttesting conditions were greatly improved over 
pretesting conditions. There were much fewer problems with 
student selection, testing facilities, and scheduling. 

Student selection was a problem only in a few instances 
where the lists of students prepared by the test and analysis 
contractor were incomplete, students were listed in the wrong 
groups or in more than one group, and lists contained students 
that did not attend the schools for which they were listed. 
For example, the majority of one district's ninth grade con- 
trol stuf;>?nts on the list provided by the test and analysis 
contractor did not attend the control school. As a result, 
no comparative evaluation of the program in the ninth grade 
was possible at this school district. 

Eight test monitors' reports showed that class sizes 
exceeded those recommended by the test and analysis contractor 
or that classrooms were overcrowded for certain grades and/or 
groups. But only one of these test monitors stated that the 
facilities were unacceptable; most stated that they were 
adequate to excellent. 

At this school district the test monitor reported that 
all experimental group students in the seventh grade were 
tested in one room (100 students). The eighth grade students 
were tested in three rooms in groups of 46, 47, and 25. With 
the exception of the group of 25, the test monitor felt that 
the test results for the first morning of evaluation testing 
were invalid because of student behavior problems. In the 
group of 25, conditions were considered marginal and would have 
been acceptable except that the junior high control students 
were tested under excellent conditions. The test monitor 
stated that this difference in the testing conditions for the 
two groups must be taken into account in comparing test re- 
sults. He also reported that "all examiners felt that the 
testing had gone a whole lot smoother than the pretesting." 
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Deviations from scheduled test dates and times were 
reported in a few instances but did not seem significant, 
with one possible exception, when one teacher allowed the 
students as much time as they needed to complete the tests. 
This teacher was replaced on the second day of testing. 

Other deviations from OEC's testing procedures occurred 
because firm personnel participated in the testing. Firm 
teachers at one school district were used as test examiners and 
at, another district, they sat in the classrooms during testing. 
Firm teachers at five other districts were called into classes 
having discipline problems and were allowed to stay until 
student behavior improved. The teachers at one of these 
school districts remained to proctor the posttesting. 

The test monitors at two school districts reported that 
the firms' elementary teachers offered their students rewards 
for behaving well and working hard during the test. At one 
of these school districts, the teachers also handed out 
candy to their students at the door of or in the test class- 
rooms. The test monitors were disappointed at the teachers' 
attempts to influence the students. At the other school dis- 
trict the teachers were also telling the elementary students 
that they would be penalized for misbehaving during the test, 
but the test monitor requested that the negative incentive 
be dropped. 

INTERIM PERFORMANCE OBJECTIVE TESTS 

Interim performance objective tests were to be used as a 
basis for up to 25 percent of the final payment to the educa- 
tional firms and as a supplemental measure of the impact of 
the individual educational firms' instructional programs. The 
tests were to be given to the experimental students five times 
during the school year to assess their mastery of the curricular 
materials to which they had been exposed. 

, Although the tests were administered, the conditions 

necessary to insure valid test results could not be met within 
the limited time before the tests were administered. As a 
result, the tests were virtually useless for evaluation pur- 
poses and questionable as a basis for payment to the firms. 

The educational firms were required to submit to the 
test and analysis contractor three times the number of items 
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(questions) required for each of the five interim performance 
objective tests that were to be administered. They were also 
to submit the curriculum objectives on which these tests were 
based and to document the relationship between each question 
and the curriculums by the first day of school. 

The test and analysis contractor then had to evaluate 
the data and determine whether the individual test items did 
or did not reflect a fair and relevant test of the educational 
firms* curriculums. If the test items were fair and relevant, 
the contractor was to certify such in writing to OEO; if not, 
the contractor was to notify OEO and the firms why they were 
not and recommend improvements, OEO*s project manager was 
to settle any disagreements between the firms and the test and 
analysis contractor. After certification, the contractor 
was to submit the questions to the school districts. The 
project directors would then randomly select one-third of these 
questions to be included on the tests, 

After'school started, OEO realized that these requirements 
could not possibly be met before the scheduled test dates. 
Consequently, the test and analysis contractor did not evaluate 
and certify these tests before they were given. Moreover, 
on the basis of its evaluation, the test and analysis contrac- 
tor concluded thai most of the tests were deficient for several 
reasons. 

Although the educational firms were requested to correct 
these deficiencies and all but one firms* tests were later 
certified as acceptable, the* certifications were of little 
value since the deficiencies were not corrected until after 
the tests had been administered. Moreover, one firm, with 
the exception of one test in one school district, never pro- 
vided interim performance objective tests for its three school 
districts. The tests were made up at the sites by each teacher; 
consequently, the tests certified were not the tests adminis- 
tered, 

OEO expected these tests to measure the students* progress 
in the reading and mathematics programs of each educational 
firm. But insufficient controls over test content and ad- 
ministration precluded such an evaluation. The pass-fail 
rate for the tests varied significantly among firms and 
school districts. 
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Kor example, in one school district, the firm administered 
1,424 mathematics tests to students in grades 7, 8, and 9, and 
the students failed only 108 of these tests, or less than 
7,6 percent. In another school district, the firm. administered 
1,374 mathemtics tests to its students in grades l\ 8, and 9, and 
the students failed 967 of these tests, or more than 70 percent. 
OEO could not determine whether a high or low pass-fail 
rate was a result of the quality of the instructional programs 
or the inappropriateness of the tests given. 
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L^lAP Tii;^ 5 

OLO's procedures in solicitins; propose Is and in awardinj; 
contracts ;,enerally did not result in effective procurcuent/ 
In its haste to be-un the experir.ient in the 1970-71 school 
year, OHO did not, in our opinion, provide a reasonable 
period of tii.ie for prospective contractors to prepare re- 
sponses to its request for proposals and for OEO to evaluate 
the responsiveness of proposals, including the reasonableness 
of cost estimates. This limited the number of companies sub- 
mitting proposals and the amount and qualitv of infornation 
in the proposals submitted. As a result, oi-0 selected some 
companies which were not financially responsible, did not 
meet the technical experience requirements, and incurred 
significant contract overruns. 

In addition, the various contracts entered into by OEO, 
the school districts-, the educational firms, and the test 
and analysis and management support contractors were defi- 
cient in that certain responsibilities were not assigned, 
were assigned to more than one contractor, and/or were in 
conflict. The OEO contracting officer and project manager 
did not carry out many of their responsibilities according 
to well-established Government policies in contract adminis- 
tration as set forth in the Federal Procurement Regulations. 

SELECTION OF THE EDUCATIONAL FIRMS 

On April 27, 1970, OEO's request for proposals 70-107 
was published in the Commerce Business Daily. The request 
announced OEO's plans to carry out a major field experiment 
in remedial education techniques in reading and mathematics 
and invited qualified firms in applied educational technology 
to submit proposals within the 2-week period ended May 11, 



According to the request, the design of the performance 
contracting experiment had not been finalized but it was 
contemplated that the firms selected would each carry out 
instructional programs in three as yet unselected school 
districts having large disadvantaged and academically de- 
ficient populations. The experimental instruction would 
be with two sets of grades, first through third and seventh 
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through ninth, for an estimated hour each dav in each subject 
for a full academic year. 



The request for proposals stated that OEO would con- 
sider only those firms with a demonstrated capability in 
educational techniques and technology appropriate for help- 
ing disadvantaged children in reading and mathematics. It 
stated furthei that the purpose of the experiment was to 
evaluate the effectiveness of existing educational techniques, 
not to develop new techniques. To be considered, a firm 
must have demonstrated its capability to begin the instruc- 
tional program by September 1970. 

The criteria for evaluating proposals were set forth 
in the request for proposals, as follows: 

"The approx. six contractors who will participate 
in this experiment will be selected upon the ba- 
sis of their responses to the following questions: 
(A) A statement of their general capability and a 
description of all corporate and staff experi- 
ences in the area of applied educational technol- 
ogy and training. (B) A full description of 
their proposed approach, I.E., the particular 
materials, procedures, types of hardware, (if 
any) and software used, etc., and a discussion 
of previous findings using this approach. (C) 
A description of how they propose to supply in- 
structional staff, I.E., whether their own in- 
structors will be supplied, whether they will 
train existing teachers or other local people, 
etc. (D) A description of the incentives, if 
any, which are part of their approach and whom 
they are mainly aimed at (E.G. Pupils, Teachers, 
parents, the school system). (E) A description 
of their approach to school/contractor coopera- 
tion, including teacher's unions." 

Thirty-one firms responded by the 14-day deadline. 
Because the request for proposals lacked specific details 
on the experiment's design, such as the size of student 
populations, OEO sent an addendum to the 31 firms on June 2, 
1970. The addendum, which spelled out in more detail the 
scale of the experiment, was prepared with the assistance 
of CEO's management support contractor and required a reply 
by June 10, 1970. 
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The firms were informed that all students would be at 
least one grade level deficient* Each firm was requested to 
indicate the final achievement levels and costs it was will- 
ing to guarantee if selected. Twenty-three of the 31 firms 
responded to the addendum. 

OEO's review of the 23 proposals was accomplished in 
two phases. The first phase consisted of grouping and rank- 
ing the firms in terms of their curriculum and hardware 
(learning systems), incentives, staffing patterns, approach 
to school-contractor relationships, and individual and staff 
experience. The phase II review was concerned with the 
minimum grade level guarantees made by the firms and the 
cost per grade level increase. 

OEO selected six educational firms which represented 
separate approaches to remedial education and which offered 
costs that would not prohibit installing the program in 
schools on a large-scale basis if warranted by the results 
of the experiment. 

OEO did not use selection criteria 
specified in request for proposals 

Our examination of the firms* proposals indicated that 
none of the firms selected had the existing educational 
techniques and demonstrated capabilities initially deemed 
necessary by OEO. Initially, the criteria for selecting 
firms, as stated in the request for proposals, was to limit 
the selection to those firms which had a demonstrated capa- 
bility in using existing educational techniques and technolo- 
gies appropriate for helping disadvantaged children in 
reading and mathematics. However, OEO's evaluation of the 
firms* proposals was not based upon this criterion but rather 
upon an assessment of a firm^s proposed innovative systems 
or approaches for helping disadvantaged students. A brief 
summary of the firms* stated experience with the approaches 
proposed to OEO follows. 

1. Firm A did not refer to any previous findings using 
its proposed instructional approach or to the speci- 
fic curriculum materials that would be used. More- 
over, the firm did not provide any information con- 
cerning its general capability, corporate and staff 
experience in applied educational technology and 
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training, or a description of its approach to 
school- contractor cooperation, including teachers' 
unions, as required by the request for proposals. 

Firm B claimed to be the Nation's largest system 
of programed learning centers with 66 in operation 
and 50 more scheduled to be opened by October 1970. 
The operating centers were testing over 1,000 stu- 
dents per month, an average of fewer than 16 stu- 
dents per center. 

Firm C, established in 1967, had prior Job Corps 
experience and was operating one learning center 
which provided instruction in mathematics and read- 
ing to over 50 students per day from preschool to 
high school. Some of these students were from 
minority groups, were dropouts, and participated in 
remedial work and enrichment study programs. 

Firm D listed its past experience as being in voca- 
tional skills training, adult basic education, and 
college remedial training and tutoring. It gave no 
indication as to the number of students involved 
and presented no discussion of previous findings, 
except for a brief statement on its college program. 
The firm stated that it was also experimenting with 
a small number of high school and junior high school 
students and dropouts using the format of the col- 
lege remedial training and tutoring program but that 
no results were available yet. 

Firm E, established in 1967, did not claim, in its 
proposal, to have had any previous experience in 
conducting remedial instruction for disadvantaged 
children. It claimed, however, to have developed 
an empirical process for creating new and modified 
materials which could be readily adopted to almost 
any subject matter. In addition, it claimed to 
have developed, tested, and refined an administra- 
tive planning tool which encouraged more precise 
design of educational planning and evaluation activi- 
ties . 

Firm F's past experience consisted of operating a 
Job Corps Center and a vocational rehabilitation 
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center. T!]c firm planned to open its first learning 
center in September 1970 , whicli would offer creative 
learnini: environments and sel f -mot i vat i on instruc- 
tional technique.-, for private students from ages 
3 to 8. 

After OHO decided not to base selection of a firm on 
the criteria set forth in the request for ]>roposals, an 
amendment to the request for proposals should have been 
issued apprising all offerors of the changes. This would 
nave notified offerors of the criteria against which their 
offers were to be measured and would have placed them on an 
equal basis as required by the Federal Procurement Regula- 
tions . 

Financial responsibility of firms 

The Federal Procurement Regulations (1-1.1203) also 
require a contracting officer to determine whether a pro« 
spective contractor (1) has adequate financial resources for 
performance or has the ability to obtain them as required 
during performance and (2) has the necessary experience, 
organization, technical qualifications, skills, and facili- 
tic:> or has the ability to obtain them. Although 01:0's con- 
tracting officer certified that the six educational firms 
were financially responsible, at least two of the firms, in 
our opinion, did not meet the conditions necessary to make 
such a determination. Financial stability was extremely 
important in this procurement because of the unusual payment 
provisions in their contracts. 

Under the contracts each firm could receive fund advances 
up to 80 percent of the total contract price in seven in- 
stallments throughout the school year. Each firm received 
advances of from about $370,000 to $655,000. Total advances 
amounted to about $3.25 million. The contracts also required 
each firm to purchase a repayment bond within 5 days of the 
effective date of the contracts. However, because i.f the 
nature of the performance incentive contract, that is, the 
uncertain amount of final payment, and the inadequacy of the 
financial resources of the firms, only one firm was success- 
ful in obtaining a repayment bond. 

On February 22, 1971, the OEO Internal Audit Division 
issued a report on its review of the financial records of the 
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six firms. The purpose of the review was to determine (1) 
the cost incurred by each firm from inception of the con- 
tracts through December 31, 1970, (2) the excess, if any, of 
the advances over costs incurred for the period ended Decem- 
ber 31, 1970, and (3) the financial status of each firm and 
its ability to refund advances in excess o{ the settlement 
claim in the event that the contract was terminated or that 
> the guaranteed student grade level gains were net achieved. 

Yhe. report concluded that four firms were cap.-'ble of guaran- 
teeing repayment of any advance in excess of the final settle- 
ment but that firms A and D were not. 

The OEO audit report pointed out that firm A was a ve.y 
small undercapitalized business operating at a loss at the 
time OEO selected it. At July 31, 1970, its books showed a rc 
tained deficit of $42,283 and curient liabilities exceeaed 
current assets by a ratio of almost 2 to 1. 

Firm D was also very small. *^tatements prepared from 
its unaudited records for the II mouths ended September 30, 
1970, indicated that it had limited financial resources and 
that income from sales other than to OEO would have been under 
$10,000 for the period. 

As a result of the failure of five of the six firms to 
obtain repayment bonds, OEO accepted alternative indemnifi- 
cation agreements in which corporRle assets were pleaged in 
lieu of a repayment bond. However, firms A and D did not 
have sufficient assets to pledge to meet the requirement. 
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SELECTION OF TEST AXP ANALYSIS COXTRACTOR 



On July 16, 1970, OEC published its request for 
proposals for selecting the test and analysis contractor in 
the Commerce Business Daily and also mailed copies to about 
50 companies. The closing date for receipt of proposals was 
July 31, 1970, 2 weeks after the date of publication. The 
scope of the work to be performed over 2 years was 
essentially to: 

1. Select, administer, and score four standardized 
nationally normed achievement tests to be used 
for payment and evaluation purposes. 

2. Review, certify, monitor, and score five interim 
performance objectives tests to be prepared by 
the educational firms and administered by the 
schools * project directors . 

3. Construct and administer a parent attitude and 
socioeconomic data questionnaire, interview ninth 
grade students, and collect other data for 
analysis . 

4. Develop a statistical analysis plan on test scores 
and other data collected. 

5. Evaluate and report results. 

The criteria and weights used by OEC in evaluating and 
selecting the contractor were: 40 points on technical 
proposal, 35 points on corporate experience, and 25 points 
on individual experience. 

OEO received nine proposals in the 2-week response 
period and evaluated them in three phases over 2 weeks. 

Phase I consisted of reviewing and ranking each 
proposal in terms of its responsiveness to the request for 
proposals. Five of the companies at this time were found 
unacceptable because each lacked corporate and individual 
experience in test and measurements, mathematics and reading 
curriculums, school management, large-scale testing and test 
scoring, and data processing and statistical analysis. The 
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request for proposals had stated that a successful bid 
would require organizational and professional staff 
experience in all the above* 

Phase II consisted of interviewing the four companies 
judged responsive to discuss every aspect of their proposals, 
to question them on their ability to manage each phase of 
the work, and to explore in detail their approaches to the 
technical issues. 

Phase III consisted of giving those companies inter- 
viewed in phase II an opportunity to correct deficiencies 
noted during phase II. 

During phase III OEO eliminated one company from con- 
sideration because its cost proposal was nearly twice that 
of any other company and it informed OEO that costs could 
not be significantly reduced. The cost proposals of the 
three others, as revised during phase III, ranged from 
$503,414 to $714,077. OEO selected the number- one- ranked 
company at an estimated cost of $614,346. 

In reviewing the proposal of the successful company, 
the OEO evaluation team commented that it was also weak in 
three of the areas which resulted in the elimination of five 
other companies during phase I. The company was considered 
weak in these areas because (1) none of the assigned staff 
had ever participated in large-scale testing and test scor- 
ing, (2) the company had limited expertise with the kind of 
quantitative analytical approach required for this evalua- 
tion, and (3) it lacked corporate experience in mass- testing. 

OEO authorized the test and analysis contractor to 
incur costs beginning on August 17, 1970. On October 30, 
1970, both parties executed a cost-plus-a-fixed-fee contract 
in the estimated amount of $614,346. The final cost to OEO 
for the services of this contractor, however, will amount to 
over $1,082,100, an increase of 76 percent over the initial 
contract amount. A greater overrun would have occurred 
except that cost savings resulting from reductions in the 
scope of the contract, as directed by the project manager, 
amounted to about $142, 000, 
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Factors contributing to 

s ign i f leant contract cost overrun 



Two factors contributed to the significant contract 
overrun incurred foi the test and analysis function. First, 
one important item of information in the request for pro- 
posals concerning the selection of schools and students was 
inaccurate. This information significantly affected both 
the proposed costs and the proposed technical evaluations 
of program results in the companies' proposals. Second, 
the cost proposal of the test and analysis contractor did 
not completely and accurately estimate the costs to perform 
the functions specified in the request for proposals. 

OEO stated in its request for proposals that the 
schools would be randomly assigned as control or experi- 
mental, that the students within the experimental schools 
would be randomly assigned to experimental and comparison 
groups, and that the students within the control scnools 
would be randomly selected from all low-achieving students 
and assigned to the control group. However, the schools 
were selected on the basis of the most deficient schools in 
the district and students selected were the most deficient 
in the schools. 

As discussed in chapter 2, the resulting mismatch of 
experimental and control groups was a significant problem 
in analyzing the results. Since the initial contract costs 
were based on a random selection process, the test and analy- 
sis contractor incurred significant additional costs in 
attempting to statistically account for the bias introduced 
througli the nonrandom selection. 

Also OEO did not determine the completeness and 
reasonableness of the cost estimates submitted by the test 
and analysis contractor, though required by the Federal 
Procurement Regulations, Consequently, a substantial portion 
of the contract overrun was due to the fact that the cost 
estimates were not a complete and accurate estimate of the 
costs to perform the tasks specified in the request for 
proposals • 

The most significant item contributing to the cost 
overrun was the omission of payments for onsite test examin- 
ers from the contractor's proposal. The proposals of two 
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unsuccessful companies indicated, however, that there was 
confusion as to who would pay the onsite test examiners* 
One proposed to pay the onsite examiners, while the other 
stated specifically that it would not pay them. 

OEO contract files contained no documentation as to 
this significant omission or to the time it became evident* 
OEO officials informed us, however, that they did not real- 
ize that the test and analysis contractoi had omitted the 
cost for this item. This cost omission was discovered some- 
time between the date the contractor was selected, August 17, 
1970, and the first day of pretesting in the school 
districts, August 31, 1970. 

Although the test administration requirements were 
significantly reduced during the school year, these costs 
exceeded $200,000. OEO informed us that it assumed a 
$50,000 item in the cost proposal provided for onsite test 
examiners. The contractor stated the $30,000 in its pro- 
posal provided only for 21 onsite test coordinators and that 
it assumed OEO or the schools would pay the onsite test 
examiners . 

Pretesting was completed by October 2, 1970, The cost 
to the test and analysis contractor for onsite test examiners 
for pretesting amounted to about $94,000. GEO did not, 
however, include an estimate of the costs for test examiners 
for either pretesting or posttesting even though the con- 
tract was not signed until October 30, 1970. 

OEO's discovery of the omission should have, as a matter 
of sound procurement practice, prompted either a reopening 
of regotiations or a formal reconsideration of the award 
selection on the basis of new information. As a practical 
matter, however, OEO's hands were tied because conducting 
pretesting within the time frames set forth in the contracts 
was critical to the experiment. However, there was no excuse 
for the failure to squarely confront the question raised and 
formally resolve it. 

The failure of OEO to discover the cost omission during 
its evaluation of proposals point? up a material deficiency 
in its procurement practices; that is, the failure to con- 
duct the cost analysis required by the Federal Procurement 
Regulations 1-3 . 807 -2 (c) . Moreover, OEO's action in not 
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attempting to cover this major cost item in its formal 
contract is questionable from the standpoint of contract 
administration. At a minimum, OEO should document its 
discovery and resolution of such situations. 

Other items not included in the test and analysis 
contractor's cost proposal, such as the cost of travel of 
test monitors for the interim performance objective tests, 
communication, and freight, totaled approximately $27,000. 
The contractor's underestimates of the costs of a number of 
other items accounted for overruns of an additional $145,000. 
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SHLECTION OF M/\NAGEMENT SUPPORT CONTRACTOR 



During April and the first part of May 1970, OEO 
intended to award the management support contract to one of 
A'O companies being considered. Both had subn^ittcd proposals 
and discussed the procurement with OEO. The OEO General 
Counsel, however, deemed this type of procurement unsound 
and recommended that competition be obtained for the procure- 
ment . 

On May 7, 1970, OEO invited seven selected companies, 
including the two with whom negotiations had already been 
held, to attend a presolici tation conference on May 12, 1970. 
At this conference, the companies were given the request for 
proposals, which had a closing date of May 19, 1970--a re- 
sponse time of only 7 days. GEO officials informed us that 
it had to contract for prior experience and therefore the 
competition was limited. 

The request for proposals set forth nine tasks that 
the companies were to address in their proposals. The tasks 
included developing criteria and a system for selecting 
schools and students, developing and implementing a documenta- 
tion system, developing a cost data system for comparing pro- 
gram component costs to student achievement, and auditing 
educational firm curriculums. 

The criteria and weights to be used in evaluating and 
selecting a company to perform the management support func- 
tion were (1) 40 points for the technical proposal, (2) 20 
points for relevant corporate experience, and (3) 40 points 
for relevant individual experience. Only three companies 
submitted proposals within the week allowed-- the two with 
whom OEO had already been negotiating and one other. One 
company that did not submit a proposal informed OEO by letter 
that it could not develop an adequate proposal in the time 
allowed. After initial evaluation of the three proposals, 
only the two companies with which OEO had originally negoti- 
ated were invited to attend conferences for further negotia- 
tions . 

The third was eliminated from further consideration be- 
cause its proposal, according to OEO, was too general and the 
company lacked corporate and individual experience in perfor- 
mance contracting. The successful management support 
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conti^actor uas selected within 7 days after the closing date 
for proposals because its technical proposal was judged 
superior in its approaches to developing and irnpleLient ing 
information systems, student selection and matching, mitiga- 
tion of testing problems, and cost analyses. 

It appears that although OEO attempted to achieve some 
competition for the award, the criteria and weights speci- 
fied in the request for proposals and used by OEO in evaluat- 
ing the proposals restricted the bidders to the two com- 
panies with which OEO initially negotiated. The request for 
proposals stated that corporate and individual experience 
in education which was relevant to the tasks to be performed 
would account for 60 percent of the company's total rating. 
In evaluating the three proposals, OEO emphasized corporate 
and individual experience in performance contracting in edu- 
cation. Since there had been only one well-known experiment 
in performance contracting, Texarkana, and the two companies 
initially considered by OEO had the only individual and/or 
corporate experience as a result of the Texarkana experiment, 
it appears that no othei^'companies could have met OEO's 
qualifications. Moreover, these two companies had an addi- 
tional advantage in that they had already submitted prelim- 
inary proposals and held discussions with OEO prior to the 
presolicitat ion conference . Consequently, the procurement 
procedures amounted to little more than token acquiescence 
by OEO program officials to the General Counsel's suggestion 
that competition be solicited. 

OTHER CONTRACT ADMINISTRATIO N 
PROCEDURES INADEQUATE 

The contracting officer and project manager did not 
carry out their responsibilities in accordance with well- 
established Government policies in contract administration 
set forth in the Federal Procurement Regulations. The 
contracting officer did not perform all required administra- 
tive actions necessary for effective contracting, and on 
numerous occasions the project manager exceeded his authority 
by issuing orders, both oral and written, changing the 
scope of the work, the compensation, and the period of per- 
formance without the written approval of the contracting 
officer. 
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Some of the deviations from the Federal Procurement 
Regulations resulted from insufficient leadtime and tiie 
need to make irjunediate decisions concerning the conduct of 
specific functions during the experiment. However, 01:0 did 
not reduce to writing changes in the scope of the work, com- 
pensation, or period of performance on a timely basis. In 
some instances the final written contract modifications 
amounted to little more than a written ratification of the 
events that took place during the school year. 

Contracting of f i cer 

Federal contracting officers are responsible for safe- 
guarding the Government's interests and insuring that 
contractors comply with their contracts. CEO's contracting 
officer, however, did not fulfill his responsibility in tjie 
following areas: 

1. A cost analysis of the proposals submitted by the 
companies competing for the test and analysis and 
management support functions was not made. 

2. Subcontracts were not formally approved at the 
time of their award, though required by the prime 
contract . 

3. Although numerous changes were made in the various 
contracts, formal contract amendments were not 
made on a timely basis. 

Project manager 

As the authorized representative of the contracting 
officer, the GEO project manager had the authority to re- 
present the contracting officer in connection with the 
operations of the contractors. However, the project manager 
was not authorized to issue orders which would change the 
scope of the work, the compensation, or the period of per- 
formance. Such authority resided with the contracting of- 
ficer and any changes made should have been reduced to a 
formal written modification to the contract. 

Numerous changes in the scope, the compensation, and 
the period of performance of the various contracts, however, 
were made at the direction or with the concurrence of the 
project manager. 
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For example, OEO*s contract with the test and analysis 
contractor was amended twice during the experiment. The 
first contract modification was dated June 30, 1971, after 
the close of the school year and contained numerous changes 
to the scope, the period of performance, and tlie compensa- 
tion. With few minor exceptions, these changes had already 
been made during the school year as authorized by OEO*s 
project manager ♦ 

The second modification was dated May 3, 1972. This 
modification also made numerous changes to the scope, the 
period of performance, and compensation, all of which had 
been authorized by OEO program officials after the first 
modification. In fact, the second modification was not 
signed by both parties until over a month after the extended 
contract termination date specified in the modification. 

Ihe two modifications resulted in a net increase in 
the total contract amount of $467,800, after savings result- 
ing from reductions in the scope, and were, in effect, 
little more than written ratifications of the actual work 
performed. 

CERTAIN RESPONSIBILITIES NOT ASSIGNE D 
OR IN CONFLICT AMONG CONTRACTS 

Because of the interrelated responsibilities of the 
various parties to the experiment, similar clauses were in 
each contract detailing specific responsibilities. A 
number of functions, however, were either not assigned, 
were assigned to more than one contractor, and/or were in 
conflict . 

These deficiencies were due, in part, to the short time 
available during the implementation of the experiment to 
iron out the complicated interrelationships of the various 
parties involved* 

Preparation and administration of 
interim performance objective tests 

The contracts with the educational firms stated that the 
test and analysis contractor would construct interim per- 
formance objective tests from the pool of test items (ques- 
tions) furnished by the firms. The test and analysis 
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contractor's contract stated, however, that the project 
directors in the school districts would prepare the tests 
from the pool of test items received from the test and 
analysis contractor. The school districts' contracts did 
not mention who would be responsible. 

With regard to test administration, OEO's contract with 
the test and analysis cortractor stated that the school 
districts would administer the tests. The contracts with 
the school districts and the educational firms stated that 
OEO or its designee would administer the tests. The educa- 
tional firms actually constructed and administered, or as- 
sisted in administering, their own in4:erim performance 
objective tests. OEO has paid or intends to pay the educa- 
tional firms a total of about $200,000 for the added work in 
preparing and administering these tests. 

Dropout and replacement testing 

The school district contracts with OEO stated that 
OEO's designee would test student dropouts. The educational 
firms' contracts stated that the test and analysis contrac- 
tor would test them. However, the test and analysis con- 
tractor's contract stated that it would not be responsible 
for dropout and replacement testing of students. The matter 
was resolved by an amendment to the contracts with the 
school districts which provided additional funds for this 
purpose of $1,250 per school district, or a total of 
$22,500. 

Curriculum audit 

OEO's contract with the managenient support contractor 
required the contractor to develop techniques to assess the 
degree to which increased learning was due to "teaching ro 
the tests" or to actual improvement in achievement level. 
According to the contracts with the educational firms, the 
test and analysis contractor would preaudit the educational 
firms' instructional programs to determine whether stand- 
ardized test items were included in the curriculums. No 
requirement for such a preaudit was contained in the test 
and analysis contractor's contract. The management support 
contractor's responsibilities were subsequently expanded to 
include a preaudit and a continuing audit of the educational 
firms' curriculums, and added compensation was provided for 
this . 
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CHAPTER 6 



PAYMENTS TO THE EDUCATIONAL FIRMS 

The crux of the performance contracting concept is that 
the educational firm is paid only to the extent that it 
raises student achievement levels. However, final payments 
to the educational firms bore little, if any, relationship 
to the initial payment provisions of the contracts or to 
the firms* success in raising student achievement levels. 

Because of (1) the insufficiency of the initial con- 
tract payment provisions, (2) changes to the scope of the 
contracts, and (3) operating conditions which hampered the 
educational firms* ability to perform, OEO made a number of 
adjustments to increase the educational firms* earnings. 
The most significant increases in the firms* earnings re- 
sulted when OEO (1) reimbursed all firms for so-called lost 
instructional time--about $845,000--and (2) dropped the in- 
centive provisions of the contracts in favor of cost reim- 
bursement for all grades in one school district and for 
three grades in another school district-- about $172,000. 

In our opinion, OEO*s adjustments go beyond the orig- 
inal language and intent of the contracts and, in some 
cases, are unreasonably generous. In making the adjust- 
ments, OEO recognized that the conditions under which the 
firms conducted their instructional programs differed sig- 
nificantly among some school districts and firms and, in 
some instances, had a detrimental effect on the firms* 
ability to perform; that is, to instruct the students and 
thereby raise their achievement levels. OEO stated that, 
although these factors had affected the firms* ability to 
meet their contract guarantees for student achievement, 
they did not relate to comparisons of student achievement 
results between experimental and control programs. 

We believe that any conditions which have adversely 
affected the performance of the educational firms may have 
also adversely affected the reliability of comparisons cI 
results between the experimental and control programs. The 
basic measure for evaluating the results of the experiment 
and for computing payments to the firms was the same-- 
student achievement. 
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Also, in settling the contracts with the educational 
firms, OEO has accepted full responsibility for any condi- 
tions not in accordance with the contract language. It 
seems to us that OEO should have expected the contractors 
to accept more of the responsibility for these conditions. 
We did not find any evidence demonstrating that the contrac- 
tors were not aware of the intent of most of the contract 
provisions at the s'art of the school year. It appears to 
us that many of the contractors* objections to OEO*s meth- 
ods for computing earnings resulted because they overstated 
their abilities in guranteeing such high increases in stu- 
dent achievement levels. 

PAYMENTS TO THE EDUCATIONAL FIRMS 
FOR PRETESTS AND POSTTESTS 

The educational firms* contracts specified that up to 
75 percent of the total contract amount could be earned on 
the basis of student achievement measured by th^ pretests 
and posttests. It is important to point out that payments 
to the firms were to be computed on the basis of each stu- 
dent* s achievement and that there was no limit on the amount 
a firm could earn per student. The 75-percent ceiling ap- 
plied to the aggregate of payments for all st dents. Gen- 
erally, a gain of 1.5 grade levels by all students would 
have been necessary for a firm to receive payment of 75 per- 
cent of the contract amount. 

During the school year OEO and the educational firms 
discovered that , because of changes to the contract scope 
and other unanticipated occurrences, the initial payment 
provisions of their contracts dealing with pretests and 
posttests were no longer sufficient. OEO and the educa- 
tional firms held discussions near and at the end of the 
school year to resolve the issues obstructing final settle- 
ment of the contracts. 

Student underenrollment 

The contracts between the educational firms and the 
school districts stated that 100 students would be provided 
in each of the 6 grades, with the exception of the 3 small 
school districts in which only 75 students per grade were 
required. However, 60 out of the 108 grades had fewer than 
the required number of students in the experimental program 
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for the full school year. Consequently, OEO devised a pay- 
ment fornuia to compensate the educational firms for these 
"empty scats/' The amount to be paid for underenrol lees , 
as proposed by OLO, was computed by taking the average pay- 
ment for full-time students multiplied by the number of un- 
derenrollecs prorated for the period of underenrollment • 

Students not below grade level 

The educational firms' contracts stated that students in 
the experimental groups would all be performing below grade 
level in mathematics and reading. There were, however, a 
number of experimental group students who performed at or 
above grade level in each school district. In one school 
district 24 percent of the experimental group students were 
performing at or above grade level. 

The educational firms argued that their programs were 
designed for underachievers and, therefore, could not be 
effective with children who were performing at or above 
grade level. Because the contract provisions had been spe- 
cific as to entry level, OEO agreed to waive the firms' min- 
imum guaranteed grade level gain and pay for any achievement 
gains made by the above-grade- level students during the 
school year. 

Students achieving lowest possible test score 

When the test and analysis contractor and OEO selected 
the standardized tests, OEO apparently overlooked the poten- 
tial problem that the measurement devices selected for first 
and second graders could not measure below a 0.6 grade 
level. Consequently, when a large segment of first graders 
and, to a lesser extent, second graders scored a 0.6 entry 
level, the educational firms felt that many of these students 
should have been ranked lower. Since there was no way to 
precisely place them on a scale between 0.0 and 0.5, OEO 
arbitrarily adjusted grade levels downward to 0.2 for first 
graders and 0.3 for second graders. 



On June 1, 1971, OEO sent the proposed contract modifi- 
cations discussed above to the firms with the stipulation 
that no payment test results would be released until the 
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modifications were signed. During the subsequent negotia- 
tions the educational firms and OEO negotiators were denied 
access to both the evaluation and payment test results* By 
late fall, however, OEO stated that this was longer 
ticable for OEO negotiators, but that b> Lhen their bargain- 
ing position was fairly well fixed and not affected by the 
knowledge of test results* The firms signed the modifica- 
tions between June and November and payment results were re- 
leased to them* Other minor modifications were also made a 
part of the contracts at this time. 

The initial payment test results released to the firms 
showed that, on the basis of the pretest and posttest scores, 
the firms earned an average of only 33 percent of the total 
possible, or $1.1 million of a possible $3.3 million. (See 
app* III.) With the excej "ion of one firm in one school 
district which earned 56 percent, none of the other firms* 
earnings exceeded 43 percent of the contract amount. Two 
lirms each earned only 19 percent in one of their schoci 
districts. 

Because of these disappointing earnings, the educational 
firms further protested to OEO that their ability to perform 
was seriously hampered during the year and consequently their 
total earnings were less than expected. Discussions were 
subsequently held to further negotiate final settlement of 
the contracts. 

Lost instructional time » 

Two educational firms had brought the issue of lost in- 
structional time to OEO*s attention during the meeting held 
before the close of the bchool year, OEO took the position 
that lost time was not an issue since the time claimed as 
lost was not significant and would have little or no impact 
on student achievement. Consequently, OEO did not include 
an adjustment for lost instructional time in its June 1 pro- 
posed amendments to the contracts. After payment test re- 
sults were released, the educational firms again argued that 
the lost instructional time contributed to the students* 
poor showing on the te^ts. 

The educational firms complained to OEO that the in- 
structional time afforded them during the school year was 
not in accordance with the contract. The rujority of the 
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contracts contained the following clause on the amount of 
time which would be available to the firms: 



"The project shall continue for the full 1970-71 
academic year, consisting of approximately 180 
class hours of instruction in each of reading and 
math." 



Some of the contracts, however, used the term "class 
periods" instead of "class hours." The firms claimed that 
the time available was less than 180 classes of 1 hour each 
per subject due to (1) testing requirements, (2) interrup- 
tions, such as assemblies and fire drills, and (3) class 
hours or periods of less than 60 minutes in some schools. 

The firms argued that the students would have achieved 
greatei- gains with more time and thus requested, during 
meetings with OEO, that they be compensated for this lost 
time. Their position was that OEO should adjust the grade 
gains on the basis of the assumptions that (1) the contract 
entitled them to 180 class periods during the year regardless 
of other provisions in the contract, such as time set aside 
for testing, and (2) the contract entitled them to 60 min- 
utes per class period. 

UEO agreed that the firms actually had less instruc- 
tional time than it had anticipated and that some adjustment 
was in order. It devised a formula for uniformly adjusting 
each firm^s earnings, as follows: 



Adjusted 

grade = Enrollment period 

gain Actual average at- 

tendance of all 
full-time students 



Expected minutes Actual 

X in class hour x grade 

Actual minutes gal/ 
in class hour 



The formula adjusts actual student achievement in di- 
rect proportion to the amount of instructional time lost. 
As we pointed out earlier in the report, research findings 
indicate that the length of a class period can have an effect 
on achievement. However, the basic assumption underlying the 
formula is that student achievement increases in direct pro - 
portion to the minutes in the class. Although there is no 
educational support for this assumption, OEO proceeded on 
this basis for the purpose of adjusting the earnings of the 
firms. 
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The results of the formula are demonstrated in the fol- 
lowing example using a hypothetical student and the actual 
adjustment formula for one firm in one school district. A 
student, John Doe, scored a grade gain of 0.9 on the pretests 
and posttests. Assuming the firm guaranteed a minimum 
1.0 grade gain, no payment was due. The school district in 
which John Doe was a student, however, had instructional 
classes of only 40 minutes and the average student attendance 
during the year in this school district was 134 days. The 
enrollment period for the school year was 165 days. John 
Doe's adjusted grade gain would be 1.7 computed as follows: 

165 X 61O X .9 = 1.7 
134 40 

The firm would then receive $81 for achieving the min- 
inum guaranteed grade increase for John Doe of 1.0 and 
S8.25 for each 0.1 grade level increase above the minimum, 
or $57.75. Total payment to the firm for John Doe would be 
$138.75 based on pretests and posttests. 

Enrollment period 

Although the firms' contracts stipulated that approxi- 
mately 180 class hours, or periods, would be available, two 
otner clauses in the contracts set aside a pe'-iod of up to 
25 days during which testing would be conducted, as follows. 

1. OEO or its designee shall administer the pretests 
not more that 10 days after the contractor's first 
day of classes. 

2. OEO or its designee shall administer the posttest 
no earlier than 10 days prior to the contractor's 
last day of classes. This was later changed 1 ^ 
15 days so as not to have testing going on durinp 
the last week of school. 

OEO officials stated that the contracts were clear on 
the number of days that would be set aside for posttesting 
and concluded thac the educational firms should not have 
expected students to achieve further grade gains after the 
posttesting began. OEO felt that the clause on the period 
during which pretesting would be conducted left unclear the 
number of days, if any, that the firms should expect to 
lose because of testing. In effect, OEO interpreted the 
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contract to mean that the 180 days specified in the con- 
tracts less 15 days for posttesting, or 165 days, would be 
available for instruction. 

On the basis of the contract language, it appears fair 
to us to say that the firms assumed the risk that, at the 
worst, pretesting would be completed on the 10th day after 
the start of school and posttesting would commence 15 days 
before the end of the school year. It would not have bene- 
fited the firms to instruct students before the pretest 
since theoretically it might enhance the students* initial 
scores to the economic detriment of the firms. By the same 
token , students * achievement resulting from ins tx^uction 
after the posttest would not be recognized for payment 
purposes. It appears to us that the contract clearly in- 
dicated that pretesting and posttesting was independent of 
class instruction and that the firms would not have a full 
180 class hours for instruction because of pretesting and 
posttesting requirements . 

Class hour or period 

OEO used the 165 days as the numerator in the first 
fraction, the enrollment period, and 60 minutes as the num- 
erator in the second, expected minutes in each class hour. 
OEO stated that the meaning of class hours, or periods, in 
the firms* contracts was also ambiguous but that it initially 
intended class hours, or periods, to be 60 minutes. 

The denominator in the first fraction, actual average 
attendance, was computed on the basis of the actual average 
attendance of full-term students. (See app. IV.) The 
denominator in the second fraction, actual minutes in each 
class period, was, as the term implies, the actual length 
of the class period in minutes, calculated by averaging 
the class periods for grades 1 to 3 and 7 to 9 . Moreover, 
when actual class minutes exceeded 50 minutes, OEO used 50 
as the denominator. We believe that in calculating the 
average attendance and actual class minutes, OEO has taken 
a position which is not reasonable or equitable to the Gov- 
ernment, as shown in the following examples. 

Example l -^Although OEO reduced the enrollment period 
from 180 to 165 days by eliminating the last 15 days of the 
school year because of posttesting, it also reduced the 
actual average attendance to a maximum of 155 days because 



66 



OEO did not consider student attendance during the first 
10 days of school, the period during which pretests were 
to be administered. By disregarding student attendance 
both during the first 10 days and the final 15 days, a 
maximum of 155 days remained for computing student attend- 
ance and OEO automatically compensated the firms for 
10 days. In addition, OEO has reimbursed firms for evrery 
student absence during the year which, in effect, is the 
same as if OEO had guaranteed absolute perfect student 
attendance. The firms* earnings were adjusted upward to 
the extent that actual student attendance during the year 
was less than perfect. 

The use of actual average attendance is inconsistent 
with a provision in the firms* contracts which stated that 
students may not be dropped from the program unless absent 
for 10 consecutive days or intermittently for 15 days over 
3 months. OEO apparently disregarded the implications of 
this provision by regarding normal student absences as a 
condition for which the firms should be reimbursed. Within 
the parameters of the contracts, the firms clearly bore 
the risk of absences during the year. 

Example 2 -'A class hour or period varied significantly 
in the school districts, and, more often than not, it was 
less than 60 minutes. Since OEO admitted to the firms that 
it had contemplated 60 minutes of class time per subject 
per day, it agreed to reimburse the firms for the lost in- 
structional time. However, in adjusting the firms* earn- 
ings, OEO used 50 minutes as the maximum time available to 
the firms when, in fact, six school districts had longer 
class periods. In two school districts the firms had 
60-minute class periods, as shown in the following table. 

Actual minutes Number of 

for class period school districts 



40 


2 


42 


1 


45 


2 


48 


2 


50 


5 


54 


1 


55 


1 


56 


1 


57 


1 


60 


_2 


Total 


18 
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If the purpose of the formula was to compensate firms 
for the instructional time lost and, consequently, the loss 
in student achievement, then no adjustment for the length 
of class periods was warranted for two school districts and 
01:0 cvurcorapensated four other firms by using 50 minutes as 
the naximum. It appears then that the firms in these six 
school districts had the benefit of the extra time and still 
received a generous adjustment to their earnings. 

The net effect of the adjustment was to increase the 
overall earnings of the firms based on pretests and post- 
tests by about 78 percent, or $844,991. Each firm*s earnings 
in each school district increased at least 36 percent, and 
one increased by as much as 101 percent. (See app. V.) Al- 
though the lost time issue was one of the major obstacles to 
settlement with the educational firms, only one firm has 
settled in all three of its school districts on the basis of 
this adjustment. Two other firms liave accepted the formula 
as a basis for settlement in two of their three districts 
but have settled for nothing less than a cost reimbursement 
for all or part of the grades in their third districts. 
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PAYMENTS TO THE EDUCATIONAL FIRMS FOR INTERIM 



PERFORMANCE OBJECTIVE TESTS 



The educational firms' contracts specified that up to 
25 percent of the total contract amount could be earned on 
the basis of student performance on interim performance 
objective tests. For the firms to recover the full 25 per- 
cent, however, it was necessary for all students to pass all 
five tests in each subject. 

As pointed out previously, the requirement for interim 
performance objective tests was impractical in light of the 
short time available to the educational firms and the test 
and analysis contractor to fulfill the contract requirements. 
However, the contracts with the educational firms provided 
for payment on this basis and OEO reimbursed the firms to 
the extent that students they instructed attained passing 
grades on the tests. The firms* earnings amounted to about 
$795,000 of a possible Si, 093, 000 , or 73 percent.. (See 
app. VI.) OEO ignored, in contract settlement, contract 
clauses dealing with the dates and periods for administering 
the tests, regardless of whether the firm observed such 
requirements during the year. 

Two modifications to the original contract affected 
provisions for interim performance objective tests. The 
first provided a method of payment to the firms for students 
that were absent through no fault of the firm during the 
period in which the test was administered. The second 
modification provided for reimbursement of the costs in- 
curred by the firms in preparing and administering interim 
performance objective tests, a task not specified in the 
original contracts. 

The second modification was a part of the final con- 
tract settlements with the firms and had been signed by only 
three firms as of March 28, 1973. Payments to these firms 
for preparation and administration ranged from $29,470 to 
$38,750, an average of $33,585 per fiim. The three remain- 
ing firms will be reimbursed fcr these costs as part of 
their final contract settlements, and we assume that these 
amounts will be similar to those already paid. The total 
cost to OEO men could exceed $200,000 for preparing and 
administering the interim performance objective tests. 
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COST REIMBURSEMENT SETTLEMENTS 



As a result of extensive negotiations with the f rms , 
OEO agreed that the conditions were such during the school 
year that settlement on a cost -reimbursable basis was war- 
ranted for all grades in one school district and for three 
grades in a second school district. Under the cost- 
reimbursement settlements, the Initial contract provisions 
which called for a 75-25 percent split between pretests and 
posttests and interim performance objective tests no longer 
applied. The firms were paid strictly on the basis of costs 
incurred up to a maximum of 100 percent of the total contract 
amount. OEO*s position on this issue was that the contracts 
with the firms assume sufficient tranquility to conduct the 
experiment but that the actual conditions prevented the 
firms from performing under the contracts as contemplated. 
Therefore, the incentive payment provisions no longer 
applied. 

OEO stated that disruption to the programs for pretest- 
ing, teacher strikes, and teacher hostility were all present 
to some degree at the two school districts. Further, OEO 
concluded that to perform, the firms required an opportunity 
to improve students* skills in reading and mathematics; how- 
ever, the disruptions and prob''eins made it impossible for 
the firms to have that opportunity. 

In its overall evaluation of the results, however, OEO 
discounted the disruptions as having an adverse effect since 
equal weight was given in the analysis to the test results 
of these two school districts. 

Specific instances of the disturbances to the firms* 
programs included late testing, class disruptions from 
teachers* union opposition, a 3-day shutdown because of a 
teacher strike, classroom furniture not in place, open 
hostility between firm and school personnel, breakage and 
stealing of equipment, and severe student discipline prob- 
lems. Although the contracts made provisions for a number 
of these occurrences, OEO felt that it was necessary to 
disregard these provisions to make an equitable settlement. 
For example, specific provision was made for strikes endur- 
ing for 30 days or more but no recourse was provided to the 
firms for strikes of lesser duration. 
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As shown in the following table, payments to one firm 
were significantly greater in the school district for which 
settlement was made on a cost-reimbursable basis than ir 
the school districts for which settlement was made on the 
incentive basis, even after the payment adjustrnoats discussed 
in the previous sections were made, 

Payment to educational firm B 

Based on 

School actual student Adjusted by 

district ach ieveme nt "165 formula" Final 

1 $132,186 $171,675 ^$191,050 

2 130 ,648 217,847 ^226 , 551 

3 121 ,979 180 ,758 ^228, 000 

$384 ,813 $570 ,280 $7 Qj^j01 , 

a 

Includes approximately $20,000 cost reimbursement for 
preparing and administering interim performance objective 
tests in each school district. 

^Cost-reimbursable basis setclement. 



The two educational firms that commented on our draft 
report stated that they had sustained high financial losses 
and injury to their reputations as a result of their partici- 
pation in OEO^s experiment. The firms expressed the opinion 
that the student achievement data did not provide an ade- 
quate basis for the conclusions published by OEO or for 
payment to the firms. The firms pointed out that because of 
the many operating difficulties, primarily insufficient 
startup time, the unsuitability of the tests selected by CEO, 
and poor testing conditions, they have not been equitably 
compensated by GEO^s payment adjustments. 
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CHAPTER 7 



CONCLUSIONS AND AGENCY COM>!ENTS 

CONCLUSIONS 

Because of a number of shortcomings in both the, design 
and implementation of the experiment, the question as to 
the merits of performance contracting versus traditional 
educational methods still remains unanswered. 

The impact of factors wnich were not under study but 
which have affected the outcome of the experimental programs 
could have been minimized if OEO had provided sufficient 
time to (1) develop a good experimental design, (2) imple- 
ment the experiment in accordance with that design, and 
(3) properly acquaint and obtain the support of those indi- 
viduals and groups th-.^ were critical to the successful 
conduct of the experiment. 

The experimental design should also have been flexible 
enough to redirect or terminate further efforts when criti- 
cal design criteria were not met during implementation. 
Moreover, the assignment of specific and clear responsibili- 
ties would have minimized the conflicting roles among 
program participants. 

OEO*s decision to proceed with the experiment in light 
of the many assumptions that had to be made to meet the 
short leadtime schedule proved to be extremely costly in 
terms of the objectivtjs that were not met and the possible 
compromise to the integrity of OEO^s overall conclusion. 

As part of its overall assessment of the impact of the 
experiment, OEO initially intended to report the results of 
the experiment on a school-district-by-school-district basis 
by use of comparisons between the experimental and control 
programs and among the programs of the six educational firms. 
The experiment was designed to provide data ^or such an 
analysis. OEO did n^t report the results of the programs 
in this manner because it found no significant differences 
between the results of the majority of the programs and 
because it wat unable to determine the cause of apparent 
successes and failures on an individual-school-district 
basis. Consequently, much of the data collected on this 
basis was of little or no use. 
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In addition, OEO's nonobservance of Federal Procurement 
Regulations and otherwise questionable actions involved in 
selecting contractors and administering and settling con- 
tracts indicated the existence of serious weaknesses in 
OEO's procurement practices. 

If OEO had performed a cost analysis of the prospective 
contractors' proposals, that is, related the contractors' 
proposed costs to the tasks required in the request for 
proposals, OEO would have identified costs which were low 
in relation to the requirements and requirements for which 
no cost estimates were provided. 

OEO should have explored the financial stability of all 
the educational firms before committing itself to large fund 
advances without any assurances that the firms could repay 
advances in excess of earnings at the end of the contract 
period. In the case of one firm, final settlement negotia- 
tions apparently resulted in a lessening of the uncollec- 
tible advances by increasing the proposed payments to that 
firm. 



The President's fiscal year 1974 budget contains no 
direct appropriations to OEO and provides for the transfer 
of certain OEO programs to other Federal agencies. Funds 
will be provided in the fiscal year 1974 budgets of these 
Federal agencies for continuing these programs. OEO's re- 
search and development activities in education will be 
transferred to the National Institute of Education, 
Department of Health, Education, and Welfare. 

Because of the proposed discontinuation of funding of 
OEO as of June 30, 1973, we are not making any recommenda- 
tions to OEO for future projects of this nature. We believe, 
however, that many of the observations and conclusion, n 
this report will be of value to the Institute and local 
education authorities if similar experiments are funded in 
the future. 

AGENCY COMMENTS 

OEO's Acting First Assistant Director stated in his 
letter of April 6, 1973 (see app. I), that OEO's final report 
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in June 1972 contained a comprehensive analysis of the 
results of the experiment and that many of the problems 
pointed out in our report were appropriately noted in OEO's 
report. He stated further that OEO believes that its report 
provides a useful perspective within which the overall 
performance contracting experiment may be judged • 
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CH>\P'r KR ,8 
SC OPE OF RhVIEW 

The objective of our review was to determine (1) whether 
the design of the experiment was such that the final results 
would be useful and appropriate for making decisions on the 
feasibility of performance contracting on an expanded basis 
and (2) whether the controls instituted and exercised by 
OEO over the various contracts were adequate to insure the 
validity of the results of the experiment. 

Our review was made at OEO headquarters in Washington, 
D.C., and at the main offices of the test and analysis and 
management support contractors. We also visited 8 of the 
18 school districts to observe the operations of the ex- 
perimental instructional programs. 

Our work included: 

--Considering the findings reported by the OEO internal 
auditors and the scope and nature of their audit work 
performed at OEO headquarters and contractors' offices. 

--Reviewing the applicable legislation; OEO and other 
contracting regulations; contracts and subcontracts; 
and reports, correspondence, and other records 
pertaining to the experiment. 

--Interviewing officials of OEO, the test and analysis 
and management support contractors, the school dis- 
tricts, and the educational firms. 

Two consultants in education assisted us: 

1. Dr. Joseph Froomkin, a consultant from Washington, 
D.C. He was formerly with Office of Education, 
Department of Health, Education, and Welfare. 

2. Dr. Stephen Kleia, Director, Educational Evaluation 
Associates, Los Angeles, California. 
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APR 6 1973 

Mr. Morton E. Henlg, Associate Director 
United States General Accounting Office 
Manpower and Welfare Division 
Washington, D. C. 20548 

Dear Mr. Henig: 

Thank you for sending me the draft report on the Office of Economic 
Opportunity s Performance Contracting Experiment. 

After careful consideration we have determined that most of the 
substance of your comments, in the report which are significant were 
furnished to the GAO as early as 1971 and were published in OEO's 
final report on the experiment in June 1972 (OEO Pamphlet 3400-6). 
The OEO final report contained a comprehensive analysis of the results 
with appropriate notes about applicable statistical and methodological* 
problems, and lengthy papers on significant testing, cost and contractual 
issues. In addition, to provide a complete view of the experiment, OEO 
included statements by the scnool district representatives and four of 
the six participating educational technology companies. The GAO draft 
fails to take cognizance of the material in the OEO report and adds 
little that would help assess the validity of the experimental 
f indi.ngs. 

The draft is devoid of any reasonably constructive tone in the discussion 
of its findings. Despite its extensive criticism of OEO's procedures, 
GAO did not make any recommendations for the future conduct of projects. 
We believe the reluctance of GAO to make any recommendations based upon 
Its findings obscures both the character of the underlying events and 
the complex nature of their remedy. Since a primary purpose of an 
agency review of a GAO draft is to consider utilization of the 
recommendations, it is difficult to realize this benefit in this 
case. As stated in your r^.port, this omission was based on the grounds 
that GEO as an agency wilj. cease to exist after June 30, 1973. 
Apparently, your agency didnot take into account that OEO's research 
functions will be transferred to othf.r Federal agencies and officials 
there would presumably benefit from any such recommendations. 

It is our expectation that GAO will include in its final report, in 
addition to this letter, a copy of the previously mentioned OEO final 
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1 

report {crpy attached). We believe this report replies adequately r 
the problems noted in the GAO draft and provides a useful perspective 
within which the overall experimental effort may be judged. Also, we 
are attaching for inclusion the view of another authority. Dr. Ellis Page 
of the University of Connecticut, who calls the experiment possibly 
the most impressive ever conducted in education. 2 



[See GAO note 3.] 




Enclosures 



GAO notes: 

^Copy not attached. Copies may be obtained for $3.00 
from the National Technical Information Service, U.S. 
Department of Commerce, Springf lelc^ , Virginia 22151 
(An Experiment in Performance Contracting, PB206793, 
June 1972) . 

^Copy not attached. See Ellis B. Page, *'How We All 
Failed at Performance Contracting,*' Phi Delta Kappan , 
(October 1972). For ^n example of an opposing viev;, 
see John K. Miller, "Not Performance Contracting But 
the OEO Experiment Was a Failure,** Phi L .Ita Kapp an, 
(February 1973) . 

^Mate^ial deleted pertains to comments on matters that 
did not concern the contents of this report. 
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.WllVDIX 11 



.-NTS 



Cj>nt roc tors 



Tnuial 
•contract 



ft irnicpcnJcnt School Oistnct 
■^»ft, Texas 

•'.I r.I «;c».>ol District 
*' "r-'.-rf, Co:ineTt:«*ut 

'* ".i ^-i^ids PuMir SvJiools 
<* ■*.:.>) ids, u;c ,:jjin 

"'-il '* ;inty *:c:ioo: Board 
*» '^'JivUIc, Florida 

si-'Td City Sc».c.l Jistrtct 
' • "^^t Indiana 

^t^rx^ S..hool Piitrict .Vo. 9 
Sronx, Vcw York 

l^illas r-JcFe^c»er»t School District 
PaMas, Texas 

^•»'ool V'-iinistratn- District So. 
!^o« ^i.tnd, Maine 

/anchorage Borough School District 
Anchor«|o. Alaska 

Clarke County School District 
Atf»cn$, Georgia 

♦cNairy County School District 
Se««er, Tennessee 

United School District No, 259 
WicUta, Kansas 

Seattle School District 
Seattle, Washington 

Ptrtland School District 
Portland, Maine 

McCo»b Separate School District 
^fcCoBb, Mississippi 

Fresno City Unified School District 
Fresno, California 

Clark County School District 
Las Vegas, Nevada 

The School Distrirt of Philadelphia 
PhilO'felphia, Pennsylvania 

Total 

EDUCATIONAL FlRMS- 
A 

B 

C 
D 
C 
F 

Total 

MANAGEMENT SUPPORT CONTRACTOR 
TEST AND ANALYSIS CONTRACTOR 
PAYMENT COMPUTATION CONTRACTOR 
Total 

■indicates fmal payienti. 



142,464 
S4,JC0 
54,S28 
53,796 
47.417 
47,211 
»1,«32 
58,970 
44,191 
51,900 
60,000 
44, IM 
39,885 
59,015 
58,744 
S6,291 



tctal 



* 90.7'1 $ 86,074 
140,573 144,779 



144,074 
*49,850 
*S9,72<| 
57,721 
42,975 
49,070 
*79,851 
*62,921 
*44,942 
*51,500 
55,715 
48,868 
*4 1,751 
*60,637 
^61 ,525 
50.842 



1x18 6,052 1.192.821 

513,000 310. 63S 

864,000 *70S.601 

776,200 559,5-^7 

726,300 *601,384 

771,000 523,712 

720.000 ^06,r 5 

4_,37O.S00 3.099.042 

S26.419 S47.419 

.-6L 4_,346 1,082.153 

— ^3,000 *26.000 

%^MS^2XL t i.947.43S 
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APPENDIX III 



SCHEDULE COMPARING EDUCATIONAL flRMS' r^RSJNCS 
W^SLD OS ACTUAL STUDENT ACHlLVZMENT AS hbASURED 
BY PRETESTS AND POSTTtSTS WITH KXXIMUM TARKABLE AMO«JNT 
UNDER CONTRACT 



Educational firms 
and school districts 

FIRM A: 

Taft Independent School District 
Taft, Texas 

Hartford School District 
Har.f'irfS* Connecticut 

Grand Rapids Publ'.c Schools 
Crami taptds, MichiKar. 

Total 

FIRM B: 

Duval County School Board 
Jacksonville, Florida 

Hauond City School District 
■laaoond, Indiana 

Bronx School District No., 9 
Bronx, New York 

Total 

FIRM C: 

Dallas Independent School District 
Dallas, Texas 

School Administrative District No. 5 
Rockland, Maine 

Anchorage Soroush School District 
Anchorage, Alaska 

Total 

FIRM D: 

Clarke County School District 
Athens, Georgia 

McNairy County School District 
Selaer, Tennessee 

Unified School District No. 2S9 
Wichita, Kansas 

Total 

FIRM E 

Seattle School District 
Seattle, Vashinston 

Portland School District 
Portland, Maine 

HcConb Separate School District 
VcConb, Mss^:ftssppi 

Total 

FIRM F 

Fresno City Unified School District 
Fresno , Cal i forma 

Clark County School District 
Las Vegas, Nevada 

The School District of Philadelphia 
Philadelphia. Pennsylvania 

Total 

Total 



Maxiaua earnable jsount Earnings bar^d on 



based on pretests 
and po>ttests 



S 114.750 

135,000 

135,000 
314.750 



216. OCO 



216.000 
64B.0O0 

119.000 

119.00: 

204.150 
512.150 

111,573 

1S1,575 

IB1.575 
544.725 

212,150 

191,000 

167.400 
571.250 

180,000 

180,000 

180.000 
S40.000 



act'ial stu cnt 
achievenent 



44.691 

45,569 

56.742 
1^7,0 02 

74,192 

76,769 

73,291 
224.252 

^6,174 

«l,14l 

77.641 
194,945 

66,413 




21B.473 

64,508 
78,288 



194.612 

38,330 

53.141 

34.995 
126>466 



Percent of 
earnings to 



38.9 

33.8 

33.0 



35.5 

33.9 
34.6 

19.1 
42.9 

38.0 

33.5 

36.6 

5S.7 

28.0 
40.1 

30.3 
39.5 



51,816 31.0 



31.0 
33.6 



21.3 

29.5 

19.4 
23.4 

33. 1 
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APPENDIX IV 



ACTUAL AVF.RAGE STUDENT 
ATTENDANCE BY SCHOOL DISTRICT 



Average attendance 
in days for 

S chool district Full-time students 

McNairy County School District 

Selmer, Tennessee 142.1 
Dallas Independent School District 

Dallas, Texas 142.3 
Clark County School District 

Las Vegas, Nevada 141.9 
Anchorage Borough School District 

Anchorage, Alaska 142.3 
Clarke County School District 

Athens, Georgia 125.5 
Unified School District No. 259 

Wichita, Kansas 144.8 
Taft Independent School District 

Taft, Texas 144.1 
McComb Separate School District 

McComb, Mississippi 135.6 
Seattle School District 

Seattle, Washington 141.8 
Grand Rapids Public Schools 

Grand Rapids, Michigan 15*1.2 
Hartford School District 

Hartford, Connecticut 111.8 
Duval County School Board 

Jacksonville, Florida 145.7 
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APPENDIX IV 



Average attendance 
in days for 

School district full-time students 

School Administrative District No. 5 

Rockland, Maine 145,2 
Hammond City School District 

Hammond, Indiana 134,2 
Portland School District 

Portland, Maine 133,9 
Fresno City Unified School District 

Fresno, California 141.5 
The School District of Philadelphia 

Philadelphia, Pennsylvania 148.1 
Bronx School District No. 9 

Bronx, New York 118.0 
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APPENDIX V 



SCI«:.D';LL comparing educational FIRMS' 
AD?'.IST!:D CAANISGS IklTH EARNINGS BASED 
0\ XCT'JAL STUDENT ACHIEVLMENT AS 
MEASURED BY PRETESTS AKD POSTTESTS 



Educational firas 
and school districts 

FIRM A: 

Taft Independent School District 
Taf:, Texas 

Hartford School District 
Hartford, Connecticut 

Grand Rapids Public Schools 
Granc Rapid>, Michigan 

Total 

FIRM B. 

Duval County School Board 
Jacksonville, Florida 

Haitsonc Citr School District 
^'aTioad, Tidiana 

Bronx School District N'o. 9 
Srony, Nev fork 

Total 

FIK4 C. 

Dallas Independent School District 
Dallas, Texas 

School Administrative District '>o. S 
Rockland, Maine 

Anchorage ficrou^h School District 
Anc.iorase, Alaska 

Total 

FITM D: 

Clarke County School District 
Athens, Georgia 

HcSairy County School District 
Selaer, Tennessee 

Unified School District No. 259 
Kichita, JCa sas 

Total 

FIRM t: 

Seattle School District 
Seattle, Washington 

Portland School District 
Portland, Maine 

McCoatb Se;>3rate School District 
McConb, Mississippi 

Total 

FIc" F: 

Fresno City Unified School District 
Fresno, California 

Clark County School District 
Las Ve^as, Nevada 

The School District of Philadelphia 
Philadelphia, Pennsylvania 

Total 

Total 



Earnings based 
on actual student 
achieveaent 



S 44.691 

4S,S69 

36.742 
127. QQ2 

74,192 

7C,769 

73.291 
224.252 

36,176 

11,148 

77,641 
194. 96S 

66,413 
101«217 



64,501 

78,218 

S1.SI6 
194,612 

38,330 

S3, 141 

34>99S 
126.466 



Earnings based on 
adjusted student 
achieveaent 



$ 60,768 
12,243 

S2,061 



Percent of 
increase of 
adjusfent 



36.0 



19S.072 

113,681 

153,297 

132.069 
399.047 

67,618 

133,651 

I4Z,720 
348.989 

133,796 



90.278 
42S.6S7 

110,842 

134, 77S 

94.2S6 
339.873 

''0.613 
88,216 

63.294 

_222^J 
S I. 930. 761 



bO.S 

41.7 
S3. 6 

53.2 

99." 

SC. 2 
77.9 



70.9 

S3. 8 
79. C 



93.2 

94.8 

71.8 

72.2 

82.0 
74.6 

84.2 

66. C 

80.9 
75 6 
77.8 
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APPENDIX VI 



'>{.mi>l/LC COMPARING hDUCATIONAL FIRMS' 
LAksisCS BASED ON STUDENT ACHIEVEMENT 
M»ASUREU BY INTERIM PLRFORMANCC 

owirrivt TESTS wini maximum earnable amount 

UNDER CONTRACT 



Educational firas 
and school districts 

FIRM A. 

l»ft Independent School District 
Taft, Texas 

Hartford School District 
Harctord , Connecticut 

Grand RapiJS Public SchooU 
Grand RapiJs. Michigan 

Total 

FIRM B: 

Du¥al Countv School Board 
Jacksonville. Florida 

HaMond City School District 
Haaaond. Indiana 

Bronx School Di'trict No. 9 
Bronx, Mew Yor^i 



FIRM C: 

Dallas ladependaat School District 
Dallas. Texas 

School AdaiBistrative District No. 5 
Rockland. Maine 

Anchorage Borough School District 
Anchorage. Alaska 

Total 

FIRM D: 

Clarke County School District 
Athens. Ceoriia 

McNairy County School District 
Selaer. Tennessee 

Unified School District No. 259 
Wiciita. Kansas 

Total 

FIRM b: 

Seattle School District 
Seattle. Mashincton 

Portland School District 
Portland, Maine 

McCoab Separate School District 
McCoab. Mississippi 



^^etno City Unified School District 
l-resno, California 

'larV County School Distrm 
-is .t^A>, Nevada 

Thr ^c.'.ncl District of Ph 1 1 Jor I |>h »a 



Tot 4 



Naxiau* earnable aaouflt 
based on mteria 

perfo raance 
objective tests 



S 5t.2S0 
45.00C 
45,000 



121^250 



216.000 

63,000 

65»000 

6t.050 
194.050 

60.525 

60.525 

60.525 
ltl.575 

70.950 

66»000 

55. BOO 
_ 192. 7S0 

60.000 

60»000 

60.000 
180.000 



Larnin|s based 
on tast results 



S 26.100 

26.922 

2t.956 
tl.97t 

57.994 



160.561 



54,681 

56.593 
167.972 

52,102 

46,517 

45.572 
14^.191 



50.511 

4 4,900 
150.2S4 

31,335 
39,051 

89.832 
$794. 7B 



Percent of 
earninis to 
aaxiaua 



59.8 



64.3 
63.9 

80.5 
74.8 

74.3 

^O.O 

86.8 

83.2 
86.6 



76.9 

75.3 
70.4 

77.3 

76.5 

80. S 
78.0 



65.1 

32.4 
49.!^ 
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APPLXDIX VII 



PRIXCIPAL OFFICIALS OF THE 
OFFICE, OF ECOXOMIC OPPORTIJXITY 
RESPONSIBLE FOR ACTIVITIES 
DISCUSSED IN THIS REPORT 





Tenure of 


O f f 1 CP 




Fron 


To 


DIRECTOR: 








llo\>rara Phillips (acting) 


Jan. 


1973 


Present 


Phillip V. Sanchez 


Sept . 


19 71 


Tan 1 Q 7 ^ 


Frank C. Carlucci 


Dec . 


1 n 7 




Donald Rumsfeld 


'lay 


1 n A n 


Dec 1970 


Bertrand M. [larding (acting) 


^^•lar . 


1 n A 0 
ly Oo 


May 1969 


R. Sargent Shriver 


uc z . 


ly 04 


Mar. 19 68 


DEPUTY DIRECTOR: 








Bert A. Gallegos (acting) 


Feb. 


1973 


Present 


IVesley. L. Hjornevik 


Oct. 


1969 


Jan 1973 


Robert Perrin (acting) 


Mar. 


1968 


Oct. 1969 


Bertrand M. Harding 


June 


1966 


Mar. 1968 


ASSOCIATE DIRECTOR FOR 








ADMINISTR^^TION (note a) : 








Thomas Wolf (acting) 


Mar . 


1973 


Present 


J. Laurence McCarty (acting) 


Feb. 


1973 


Mar. 19 73 


Ernest Russell 


Apr. 


1971 


Feb. 1973 


Robert C. Cassidy 


Sept . 


1967 


Apr. 19 71 


ASSISTANT DIRECTOR FOR PLANNING, 








RESEARCH, AND EVALUATION 








(note b) : 








Dr. Brad Hainsworth (acting) 


Feb. 


1973 


Present 


Wesley L. Hjornevik (acting) 


Oct. 


1972 


Jan. 19 73 


Thomas K. Glennan, Jr. 


July 


1972 


Oct. 1972 


John 0. Wilson 


Oct. 


1959 


July 1972 


Richard Ottman (acting) 


Jan. 


1969 


Sept. 1969 


Robert A. Levine 


Nov . 


1966 


Jan. 1969 
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APPENDIX VII 



The Office of Administration nas called Office of 
Management until June 1968. 

Prior to OEO's September 1969 reorgani /at ion, ibis office 
was called the Office of Research, Plans, PiO!;r.iiiis and 
Evaluation. 
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